[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:28:41 dist-test-slave-p29l krb5kdc[29657](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:28:41 dist-test-slave-p29l krb5kdc[29657](info): set up 2 sockets
Sep 29 20:28:41 dist-test-slave-p29l krb5kdc[29657](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 20:28:42.383975 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.003s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:28:42 dist-test-slave-p29l krb5kdc[29657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:28:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:28:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.244.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
I20260929 20:28:42.568528 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:43351
--webserver_interface=127.28.244.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:36535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:43351
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:28:42.932176 29670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:28:42.932518 29670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:28:42.932706 29670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:28:42.940315 29670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:28:42.940442 29670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:28:42.940557 29670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:28:42.940634 29670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:28:42.940718 29670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:28:42.951164 29670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:36535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:43351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:43351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:28:42.952719 29670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:28:42.954128 29670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:28:42.967909 29675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:42.967981 29676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:42.967828 29678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:28:42.968324 29670 server_base.cc:1034] running on GCE node
I20260929 20:28:42.969524 29670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:28:42.971406 29670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:28:42.972710 29670 hybrid_clock.cc:648] HybridClock initialized: now 1790713722972646 us; error 87 us; skew 500 ppm
I20260929 20:28:42.973126 29670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:28:42 dist-test-slave-p29l krb5kdc[29657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713722, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:28:42.976825 29670 init.cc:377] Logged in from keytab as kudu/127.28.244.190@KRBTEST.COM (short username kudu)
I20260929 20:28:42.978546 29670 webserver.cc:460] Webserver started at http://127.28.244.190:38185/ using document root <none> and password file <none>
I20260929 20:28:42.979331 29670 fs_manager.cc:359] Metadata directory not provided
I20260929 20:28:42.979498 29670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:28:42.979894 29670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:28:42.983247 29670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "2a54164cf6074ff29ba45a1b36fb5068"
format_stamp: "Formatted at 2026-09-29 20:28:42 on dist-test-slave-p29l"
server_key: "e2f7828cfced9e931d1cfb70c23094f0"
server_key_iv: "f5ae8e37fa68a4b0a57fd73c8148efd1"
server_key_version: "encryptionkey@0"
I20260929 20:28:42.984222 29670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "2a54164cf6074ff29ba45a1b36fb5068"
format_stamp: "Formatted at 2026-09-29 20:28:42 on dist-test-slave-p29l"
server_key: "e2f7828cfced9e931d1cfb70c23094f0"
server_key_iv: "f5ae8e37fa68a4b0a57fd73c8148efd1"
server_key_version: "encryptionkey@0"
I20260929 20:28:42.990343 29670 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260929 20:28:42.994314 29685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:42.995978 29670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 20:28:42.996186 29670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "2a54164cf6074ff29ba45a1b36fb5068"
format_stamp: "Formatted at 2026-09-29 20:28:42 on dist-test-slave-p29l"
server_key: "e2f7828cfced9e931d1cfb70c23094f0"
server_key_iv: "f5ae8e37fa68a4b0a57fd73c8148efd1"
server_key_version: "encryptionkey@0"
I20260929 20:28:42.996416 29670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:28:43.013298 29670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:28:43.018467 29670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:28:43.018815 29670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:28:43.038112 29670 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:43351
I20260929 20:28:43.038136 29736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:43351 every 8 connection(s)
I20260929 20:28:43.039662 29670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:28:43.044099 29737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:28:43.047497 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 29670
I20260929 20:28:43.047883 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:28:43.049836 29650 external_mini_cluster.cc:1428] Setting key c8dda8a6d6c7b4b93736d15ae81abeda
I20260929 20:28:43.056816 29737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a54164cf6074ff29ba45a1b36fb5068: Bootstrap starting.
I20260929 20:28:43.060221 29737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a54164cf6074ff29ba45a1b36fb5068: Neither blocks nor log segments found. Creating new log.
I20260929 20:28:43.061338 29737 log.cc:824] T 00000000000000000000000000000000 P 2a54164cf6074ff29ba45a1b36fb5068: Log is configured to *not* fsync() on all Append() calls
I20260929 20:28:43.064586 29737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a54164cf6074ff29ba45a1b36fb5068: No bootstrap required, opened a new log
I20260929 20:28:43.073704 29737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a54164cf6074ff29ba45a1b36fb5068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a54164cf6074ff29ba45a1b36fb5068" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 43351 } }
I20260929 20:28:43.074184 29737 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a54164cf6074ff29ba45a1b36fb5068 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:28:43.074327 29737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a54164cf6074ff29ba45a1b36fb5068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a54164cf6074ff29ba45a1b36fb5068, State: Initialized, Role: FOLLOWER
I20260929 20:28:43.075095 29737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a54164cf6074ff29ba45a1b36fb5068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a54164cf6074ff29ba45a1b36fb5068" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 43351 } }
I20260929 20:28:43.075398 29737 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a54164cf6074ff29ba45a1b36fb5068 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:28:43.075565 29737 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a54164cf6074ff29ba45a1b36fb5068 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:28:43.075794 29737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a54164cf6074ff29ba45a1b36fb5068 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:28:43.079401 29737 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a54164cf6074ff29ba45a1b36fb5068 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a54164cf6074ff29ba45a1b36fb5068" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 43351 } }
I20260929 20:28:43.080183 29737 leader_election.cc:302] T 00000000000000000000000000000000 P 2a54164cf6074ff29ba45a1b36fb5068 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a54164cf6074ff29ba45a1b36fb5068; no voters:
I20260929 20:28:43.080950 29737 leader_election.cc:288] T 00000000000000000000000000000000 P 2a54164cf6074ff29ba45a1b36fb5068 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:28:43.081094 29742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a54164cf6074ff29ba45a1b36fb5068 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:28:43.081868 29742 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a54164cf6074ff29ba45a1b36fb5068 [term 1 LEADER]: Becoming Leader. State: Replica: 2a54164cf6074ff29ba45a1b36fb5068, State: Running, Role: LEADER
Sep 29 20:28:43 dist-test-slave-p29l krb5kdc[29657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:28:43.082548 29742 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a54164cf6074ff29ba45a1b36fb5068 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a54164cf6074ff29ba45a1b36fb5068" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 43351 } }
I20260929 20:28:43.086210 29737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a54164cf6074ff29ba45a1b36fb5068 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:28:43.086766 29743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a54164cf6074ff29ba45a1b36fb5068 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a54164cf6074ff29ba45a1b36fb5068. Latest consensus state: current_term: 1 leader_uuid: "2a54164cf6074ff29ba45a1b36fb5068" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a54164cf6074ff29ba45a1b36fb5068" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 43351 } } }
I20260929 20:28:43.087347 29743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a54164cf6074ff29ba45a1b36fb5068 [sys.catalog]: This master's current role is: LEADER
I20260929 20:28:43.087777 29744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a54164cf6074ff29ba45a1b36fb5068 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a54164cf6074ff29ba45a1b36fb5068" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a54164cf6074ff29ba45a1b36fb5068" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 43351 } } }
I20260929 20:28:43.089351 29744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a54164cf6074ff29ba45a1b36fb5068 [sys.catalog]: This master's current role is: LEADER
I20260929 20:28:43.090171 29747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:28:43.097458 29747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:28:43.099761 29739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:43.054241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46778 (local address 127.28.244.190:43351)
0929 20:28:43.055199 (+ 958us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:43.055222 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:43.056542 (+ 1320us) server_negotiation.cc:407] Connection header received
0929 20:28:43.065949 (+ 9407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:43.065977 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:43.066478 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:43.066977 (+ 499us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:43.070316 (+ 3339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:43.072529 (+ 2213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:43.074582 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:43.075389 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:43.075635 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:43.084089 (+ 8454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:43.084110 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:43.084143 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:43.084220 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:43.090448 (+ 6228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:43.091774 (+ 1326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:43.091783 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:43.091791 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:43.092207 (+ 416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:43.093116 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:43.093121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:43.093124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:43.093943 (+ 819us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:43.097376 (+ 3433us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:43.097701 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 20:28:43.098219 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":430,"threads_started":1}
I20260929 20:28:43.111485 29747 catalog_manager.cc:1347] Generated new cluster ID: 43f7e948085f43b984e1feaaf1a7a88b
I20260929 20:28:43.111620 29747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:28:43.130165 29747 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:28:43.132179 29747 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:28:43.144963 29747 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a54164cf6074ff29ba45a1b36fb5068: Generated new TSK 0
I20260929 20:28:43.146240 29747 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.244.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab.
Entry for principal kudu/127.28.244.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab.
I20260929 20:28:43.206357 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.129:0
--local_ip_for_outbound_sockets=127.28.244.129
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:43351
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:36535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:28:43.560582 29763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:28:43.560883 29763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:28:43.560987 29763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:28:43.568526 29763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:28:43.568658 29763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:28:43.568775 29763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.129
I20260929 20:28:43.579450 29763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:36535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:43351
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.129
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:28:43.580857 29763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:28:43.582273 29763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:28:43.596848 29769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:43.597378 29768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:43.597445 29771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:28:43.603351 29763 server_base.cc:1034] running on GCE node
I20260929 20:28:43.604177 29763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:28:43.605314 29763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:28:43.606626 29763 hybrid_clock.cc:648] HybridClock initialized: now 1790713723606548 us; error 68 us; skew 500 ppm
I20260929 20:28:43.607019 29763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:28:43 dist-test-slave-p29l krb5kdc[29657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713723, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:28:43.610546 29763 init.cc:377] Logged in from keytab as kudu/127.28.244.129@KRBTEST.COM (short username kudu)
I20260929 20:28:43.612316 29763 webserver.cc:460] Webserver started at http://127.28.244.129:44291/ using document root <none> and password file <none>
I20260929 20:28:43.613121 29763 fs_manager.cc:359] Metadata directory not provided
I20260929 20:28:43.613276 29763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:28:43.613641 29763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:28:43.616868 29763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-0/data/instance:
uuid: "0011d27e97994b0bb4853a2b42b27113"
format_stamp: "Formatted at 2026-09-29 20:28:43 on dist-test-slave-p29l"
server_key: "b54139b318ee8e3bcd5ab0f283e2434e"
server_key_iv: "3e620329af2a1da15543c99ce63b327a"
server_key_version: "encryptionkey@0"
I20260929 20:28:43.617699 29763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance:
uuid: "0011d27e97994b0bb4853a2b42b27113"
format_stamp: "Formatted at 2026-09-29 20:28:43 on dist-test-slave-p29l"
server_key: "b54139b318ee8e3bcd5ab0f283e2434e"
server_key_iv: "3e620329af2a1da15543c99ce63b327a"
server_key_version: "encryptionkey@0"
I20260929 20:28:43.623565 29763 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:28:43.627651 29778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:43.629325 29763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:28:43.629534 29763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-0/wal
uuid: "0011d27e97994b0bb4853a2b42b27113"
format_stamp: "Formatted at 2026-09-29 20:28:43 on dist-test-slave-p29l"
server_key: "b54139b318ee8e3bcd5ab0f283e2434e"
server_key_iv: "3e620329af2a1da15543c99ce63b327a"
server_key_version: "encryptionkey@0"
I20260929 20:28:43.629765 29763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:28:43.642580 29763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:28:43.647518 29763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:28:43.647909 29763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:28:43.648942 29763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:28:43.650705 29763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:28:43.650836 29763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:43.651010 29763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:28:43.651093 29763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:43.685775 29763 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.129:34143
I20260929 20:28:43.685808 29890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.129:34143 every 8 connection(s)
I20260929 20:28:43.687314 29763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
I20260929 20:28:43.695336 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 29763
I20260929 20:28:43.695598 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance
I20260929 20:28:43.696290 29650 external_mini_cluster.cc:1428] Setting key 9f6b139932c4a411e7709ad8a9c86964
Sep 29 20:28:43 dist-test-slave-p29l krb5kdc[29657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713723, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:28:43.710161 29893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:43.690206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:42109 (local address 127.28.244.190:43351)
0929 20:28:43.690763 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:43.690770 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:43.691991 (+ 1221us) server_negotiation.cc:407] Connection header received
0929 20:28:43.693956 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:43.693962 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:43.694062 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:43.694185 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:43.695987 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:43.696795 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:43.698443 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:43.698864 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:43.699011 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:43.702985 (+ 3974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:43.702991 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:43.702994 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:43.703028 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:43.706480 (+ 3452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:43.707575 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:43.707580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:43.707582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:43.707711 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:43.708263 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:43.708266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:43.708268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:43.708556 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:43.708675 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:43.709695 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0929 20:28:43.709886 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":313,"threads_started":1}
I20260929 20:28:43.712064 29894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:43.690834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:43351 (local address 127.28.244.129:42109)
0929 20:28:43.691740 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:43.691793 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:43.693627 (+ 1834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:43.694460 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:43.694470 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:43.695025 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:28:43.695743 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:43.695771 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:43.696964 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:43.696969 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:43.698291 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:43.698303 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:43.699106 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:43.699116 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:43.699385 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:43.700233 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:28:43.700257 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:28:43.702722 (+ 2465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:28:43.706676 (+ 3954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:43.706690 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:43.706714 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:43.707430 (+ 716us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:43.707857 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:43.707865 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:43.707871 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:43.708144 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:43.708700 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:28:43.708711 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:28:43.709229 (+ 518us) client_negotiation.cc:769] Sending connection context
0929 20:28:43.709570 (+ 341us) client_negotiation.cc:240] Negotiation successful
0929 20:28:43.710233 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":378,"threads_started":1}
I20260929 20:28:43.713863 29891 heartbeater.cc:347] Connected to a master server at 127.28.244.190:43351
I20260929 20:28:43.714250 29891 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.28.244.130@KRBTEST.COM; defaulting to no policy
I20260929 20:28:43.715090 29891 heartbeater.cc:511] Master 127.28.244.190:43351 requested a full tablet report, sending...
I20260929 20:28:43.717803 29702 ts_manager.cc:194] Registered new tserver with Master: 0011d27e97994b0bb4853a2b42b27113 (127.28.244.129:34143)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.130@KRBTEST.COM" created.
I20260929 20:28:43.719863 29702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.129@KRBTEST.COM'} at 127.28.244.129:42109
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab.
Entry for principal kudu/127.28.244.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab.
I20260929 20:28:43.748842 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.130:0
--local_ip_for_outbound_sockets=127.28.244.130
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:43351
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:36535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:28:44.092610 29897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:28:44.092972 29897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:28:44.093149 29897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:28:44.100744 29897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:28:44.100908 29897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:28:44.101102 29897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.130
I20260929 20:28:44.111148 29897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:36535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:43351
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.130
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:28:44.112640 29897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:28:44.113960 29897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:28:44.128028 29902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:44.128058 29903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:44.128219 29905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:28:44.129309 29897 server_base.cc:1034] running on GCE node
I20260929 20:28:44.130048 29897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:28:44.131148 29897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:28:44.132354 29897 hybrid_clock.cc:648] HybridClock initialized: now 1790713724132284 us; error 73 us; skew 500 ppm
I20260929 20:28:44.132778 29897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:28:44 dist-test-slave-p29l krb5kdc[29657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713724, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:28:44.136435 29897 init.cc:377] Logged in from keytab as kudu/127.28.244.130@KRBTEST.COM (short username kudu)
I20260929 20:28:44.138068 29897 webserver.cc:460] Webserver started at http://127.28.244.130:40855/ using document root <none> and password file <none>
I20260929 20:28:44.138779 29897 fs_manager.cc:359] Metadata directory not provided
I20260929 20:28:44.138888 29897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:28:44.139186 29897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:28:44.142325 29897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-1/data/instance:
uuid: "f5fe875a8b8f4d368c4c575d024c3eed"
format_stamp: "Formatted at 2026-09-29 20:28:44 on dist-test-slave-p29l"
server_key: "a1e60887ad1fec97a16b2b97aae0ede9"
server_key_iv: "ca29ee79c4750196cd22fe084569613a"
server_key_version: "encryptionkey@0"
I20260929 20:28:44.143085 29897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance:
uuid: "f5fe875a8b8f4d368c4c575d024c3eed"
format_stamp: "Formatted at 2026-09-29 20:28:44 on dist-test-slave-p29l"
server_key: "a1e60887ad1fec97a16b2b97aae0ede9"
server_key_iv: "ca29ee79c4750196cd22fe084569613a"
server_key_version: "encryptionkey@0"
I20260929 20:28:44.150665 29897 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260929 20:28:44.154814 29912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:44.156476 29897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 20:28:44.156626 29897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-1/wal
uuid: "f5fe875a8b8f4d368c4c575d024c3eed"
format_stamp: "Formatted at 2026-09-29 20:28:44 on dist-test-slave-p29l"
server_key: "a1e60887ad1fec97a16b2b97aae0ede9"
server_key_iv: "ca29ee79c4750196cd22fe084569613a"
server_key_version: "encryptionkey@0"
I20260929 20:28:44.156813 29897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:28:44.176218 29897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:28:44.180822 29897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:28:44.181087 29897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:28:44.182042 29897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:28:44.183382 29897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:28:44.183454 29897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:44.183544 29897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:28:44.183588 29897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:44.218317 29897 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.130:43623
I20260929 20:28:44.218367 30024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.130:43623 every 8 connection(s)
I20260929 20:28:44.219674 29897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
I20260929 20:28:44.227213 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 29897
I20260929 20:28:44.227437 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance
I20260929 20:28:44.228005 29650 external_mini_cluster.cc:1428] Setting key 8bcc22ad8735c6bd8b4101bd80cac7c3
Sep 29 20:28:44 dist-test-slave-p29l krb5kdc[29657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713724, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:28:44.243149 30027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:44.222455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:58165 (local address 127.28.244.190:43351)
0929 20:28:44.222884 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:44.222889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:44.224031 (+ 1142us) server_negotiation.cc:407] Connection header received
0929 20:28:44.225986 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:44.225993 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:44.226096 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:44.226222 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:44.228512 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:44.229299 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:44.231353 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:44.232231 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:44.232411 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:44.236534 (+ 4123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:44.236541 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:44.236544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:44.236581 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:44.239386 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:44.240533 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:44.240538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:44.240540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:44.240671 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:44.241220 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:44.241224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:44.241228 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:44.241555 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:44.241674 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:44.242703 (+ 1029us) server_negotiation.cc:299] Negotiation successful
0929 20:28:44.242881 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":213,"threads_started":1}
I20260929 20:28:44.244989 30028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:44.223041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:43351 (local address 127.28.244.130:58165)
0929 20:28:44.223790 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:44.223896 (+ 106us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:44.225687 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:44.226483 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:44.226494 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:44.227273 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:28:44.228267 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:44.228328 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:44.229523 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:44.229528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:44.231183 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:44.231197 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:44.232472 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:44.232477 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:44.232700 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:44.233424 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:28:44.233454 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:28:44.236246 (+ 2792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:28:44.239600 (+ 3354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:44.239613 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:44.239637 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:44.240393 (+ 756us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:44.240791 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:44.240798 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:44.240805 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:44.241083 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:44.241695 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:28:44.241707 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:28:44.242258 (+ 551us) client_negotiation.cc:769] Sending connection context
0929 20:28:44.242635 (+ 377us) client_negotiation.cc:240] Negotiation successful
0929 20:28:44.243316 (+ 681us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":292,"threads_started":1}
WARNING: no policy specified for kudu/127.28.244.131@KRBTEST.COM; defaulting to no policy
I20260929 20:28:44.246646 30025 heartbeater.cc:347] Connected to a master server at 127.28.244.190:43351
I20260929 20:28:44.246986 30025 heartbeater.cc:464] Registering TS with master...
I20260929 20:28:44.247845 30025 heartbeater.cc:511] Master 127.28.244.190:43351 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.131@KRBTEST.COM" created.
I20260929 20:28:44.249469 29702 ts_manager.cc:194] Registered new tserver with Master: f5fe875a8b8f4d368c4c575d024c3eed (127.28.244.130:43623)
I20260929 20:28:44.250315 29702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.130@KRBTEST.COM'} at 127.28.244.130:58165
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab.
Entry for principal kudu/127.28.244.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab.
I20260929 20:28:44.271560 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.131:0
--local_ip_for_outbound_sockets=127.28.244.131
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:43351
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:36535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:28:44.622823 30031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:28:44.623185 30031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:28:44.623365 30031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:28:44.631075 30031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:28:44.631239 30031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:28:44.631423 30031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.131
I20260929 20:28:44.641928 30031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:36535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:43351
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.131
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:28:44.643362 30031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:28:44.644799 30031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:28:44.660199 30036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:44.660135 30039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:44.660218 30037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:28:44.666143 30031 server_base.cc:1034] running on GCE node
I20260929 20:28:44.667227 30031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:28:44.668854 30031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:28:44.672226 30031 hybrid_clock.cc:648] HybridClock initialized: now 1790713724672147 us; error 46 us; skew 500 ppm
I20260929 20:28:44.672724 30031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:28:44 dist-test-slave-p29l krb5kdc[29657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713724, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:28:44.676633 30031 init.cc:377] Logged in from keytab as kudu/127.28.244.131@KRBTEST.COM (short username kudu)
I20260929 20:28:44.678206 30031 webserver.cc:460] Webserver started at http://127.28.244.131:44835/ using document root <none> and password file <none>
I20260929 20:28:44.679075 30031 fs_manager.cc:359] Metadata directory not provided
I20260929 20:28:44.679224 30031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:28:44.679558 30031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:28:44.682742 30031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-2/data/instance:
uuid: "29e067f455c64b9eb504bc21c413a062"
format_stamp: "Formatted at 2026-09-29 20:28:44 on dist-test-slave-p29l"
server_key: "f905ed05bd88d832f96bc44fac490c56"
server_key_iv: "f3d199ccffd7b72937a10519ca0ec329"
server_key_version: "encryptionkey@0"
I20260929 20:28:44.683558 30031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance:
uuid: "29e067f455c64b9eb504bc21c413a062"
format_stamp: "Formatted at 2026-09-29 20:28:44 on dist-test-slave-p29l"
server_key: "f905ed05bd88d832f96bc44fac490c56"
server_key_iv: "f3d199ccffd7b72937a10519ca0ec329"
server_key_version: "encryptionkey@0"
I20260929 20:28:44.689646 30031 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260929 20:28:44.693447 30046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:44.694970 30031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:28:44.695171 30031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-2/wal
uuid: "29e067f455c64b9eb504bc21c413a062"
format_stamp: "Formatted at 2026-09-29 20:28:44 on dist-test-slave-p29l"
server_key: "f905ed05bd88d832f96bc44fac490c56"
server_key_iv: "f3d199ccffd7b72937a10519ca0ec329"
server_key_version: "encryptionkey@0"
I20260929 20:28:44.695408 30031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:28:44.723040 30031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:28:44.724272 29891 heartbeater.cc:503] Master 127.28.244.190:43351 was elected leader, sending a full tablet report...
I20260929 20:28:44.728158 30031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:28:44.728479 30031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:28:44.729532 30031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:28:44.731084 30031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:28:44.731189 30031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:44.731345 30031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:28:44.731419 30031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:44.766321 30031 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.131:37685
I20260929 20:28:44.766360 30158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.131:37685 every 8 connection(s)
I20260929 20:28:44.767732 30031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
I20260929 20:28:44.770459 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 30031
I20260929 20:28:44.770677 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance
I20260929 20:28:44.771145 29650 external_mini_cluster.cc:1428] Setting key d32fc72f97a2f218d341ee658663267c
Sep 29 20:28:44 dist-test-slave-p29l krb5kdc[29657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713724, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:28:44.791013 30161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:44.771598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:43293 (local address 127.28.244.190:43351)
0929 20:28:44.772182 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:44.772189 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:44.773468 (+ 1279us) server_negotiation.cc:407] Connection header received
0929 20:28:44.775981 (+ 2513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:44.775986 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:44.776069 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:44.776162 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:44.777799 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:44.778655 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:44.780506 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:44.781007 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:44.781109 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:44.784681 (+ 3572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:44.784686 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:44.784690 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:44.784724 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:44.787721 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:44.788691 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:44.788695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:44.788698 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:44.788820 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:44.789209 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:44.789212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:44.789215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:44.789486 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:44.789607 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:44.790364 (+ 757us) server_negotiation.cc:299] Negotiation successful
0929 20:28:44.790723 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":303,"threads_started":1}
I20260929 20:28:44.791884 30162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:44.772290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:43351 (local address 127.28.244.131:43293)
0929 20:28:44.773261 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:44.773324 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:44.775600 (+ 2276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:44.776398 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:44.776406 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:44.776982 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:28:44.777616 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:44.777635 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:44.778835 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:44.778839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:44.780356 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:44.780375 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:44.781218 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:44.781223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:44.781440 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:44.782135 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:28:44.782159 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:28:44.784452 (+ 2293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:28:44.787895 (+ 3443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:44.787904 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:44.787924 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:44.788566 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:44.788916 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:44.788919 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:44.788921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:44.789111 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:44.789591 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:28:44.789598 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:28:44.789952 (+ 354us) client_negotiation.cc:769] Sending connection context
0929 20:28:44.790196 (+ 244us) client_negotiation.cc:240] Negotiation successful
0929 20:28:44.790623 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":388,"threads_started":1}
I20260929 20:28:44.793404 30159 heartbeater.cc:347] Connected to a master server at 127.28.244.190:43351
I20260929 20:28:44.793767 30159 heartbeater.cc:464] Registering TS with master...
I20260929 20:28:44.794548 30159 heartbeater.cc:511] Master 127.28.244.190:43351 requested a full tablet report, sending...
I20260929 20:28:44.796140 29702 ts_manager.cc:194] Registered new tserver with Master: 29e067f455c64b9eb504bc21c413a062 (127.28.244.131:37685)
I20260929 20:28:44.797008 29702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.131@KRBTEST.COM'} at 127.28.244.131:43293
I20260929 20:28:44.801252 29650 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 20:28:44 dist-test-slave-p29l krb5kdc[29657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713724, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 20:28:44 dist-test-slave-p29l krb5kdc[29657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713724, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.244.129@KRBTEST.COM
I20260929 20:28:44.848008 30167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:44.831360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47348 (local address 127.28.244.129:34143)
0929 20:28:44.832118 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:44.832127 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:44.832161 (+ 34us) server_negotiation.cc:407] Connection header received
0929 20:28:44.832360 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:44.832370 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:44.832624 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:44.832796 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:44.834339 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:44.835379 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:44.836750 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:44.837353 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:44.837492 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:44.841669 (+ 4177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:44.841683 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:44.841699 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:44.841745 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:44.845107 (+ 3362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:44.845884 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:44.845893 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:44.845895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:44.846048 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:44.846451 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:44.846455 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:44.846457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:44.846847 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:44.847090 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:44.847396 (+ 306us) server_negotiation.cc:299] Negotiation successful
0929 20:28:44.847617 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":367,"threads_started":1}
W20260929 20:28:44.849393 29806 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47348
Sep 29 20:28:44 dist-test-slave-p29l krb5kdc[29657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 20:28:44 dist-test-slave-p29l krb5kdc[29657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.129@KRBTEST.COM
I20260929 20:28:44.895188 30167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:44.880688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47362 (local address 127.28.244.129:34143)
0929 20:28:44.880908 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:44.880913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:44.881002 (+ 89us) server_negotiation.cc:407] Connection header received
0929 20:28:44.881210 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:44.881214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:44.881281 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:44.881369 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:44.882794 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:44.883502 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:44.884866 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:44.885322 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:44.885461 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:44.889873 (+ 4412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:44.889879 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:44.889882 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:44.889925 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:44.892596 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:44.893280 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:44.893284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:44.893286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:44.893413 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:44.893805 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:44.893810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:44.893812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:44.894125 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:44.894333 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:44.894765 (+ 432us) server_negotiation.cc:299] Negotiation successful
0929 20:28:44.894949 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 20:28:44.897042 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 29763
I20260929 20:28:44.911060 30167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:44.898090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47368 (local address 127.28.244.129:34143)
0929 20:28:44.898296 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:44.898304 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:44.898680 (+ 376us) server_negotiation.cc:407] Connection header received
0929 20:28:44.898882 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:44.898903 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:44.898964 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:44.899060 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:44.900534 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:44.901285 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:44.902599 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:44.903021 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:44.903152 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:44.905542 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:44.905549 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:44.905558 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:44.905594 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:44.908328 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:44.909056 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:44.909062 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:44.909066 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:44.909221 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:44.909605 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:44.909611 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:44.909616 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:44.909897 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:44.910161 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:44.910461 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 20:28:44.910684 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:28:44.911877 29886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:28:44.996631 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 29763
I20260929 20:28:45.014791 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 29897
Sep 29 20:28:45 dist-test-slave-p29l krb5kdc[29657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.130@KRBTEST.COM
I20260929 20:28:45.031993 30174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:45.015705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52102 (local address 127.28.244.130:43623)
0929 20:28:45.016281 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:45.016289 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:45.016314 (+ 25us) server_negotiation.cc:407] Connection header received
0929 20:28:45.016377 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:45.016385 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:45.016614 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:45.016753 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:45.017953 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:45.018985 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:45.020373 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:45.020962 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:45.021071 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:45.025257 (+ 4186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:45.025271 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:45.025290 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:45.025332 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:45.028776 (+ 3444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:45.029546 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:45.029556 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:45.029562 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:45.029722 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:45.030154 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:45.030158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:45.030161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:45.030567 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:45.030851 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:45.031250 (+ 399us) server_negotiation.cc:299] Negotiation successful
0929 20:28:45.031579 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":288,"threads_started":1}
I20260929 20:28:45.033340 30020 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:28:45.118009 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 29897
I20260929 20:28:45.141479 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 30031
Sep 29 20:28:45 dist-test-slave-p29l krb5kdc[29657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.131@KRBTEST.COM
I20260929 20:28:45.159272 30176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:45.142642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59930 (local address 127.28.244.131:37685)
0929 20:28:45.143322 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:45.143329 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:45.143358 (+ 29us) server_negotiation.cc:407] Connection header received
0929 20:28:45.143420 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:45.143430 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:45.143650 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:45.143855 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:45.145354 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:45.146433 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:45.147732 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:45.148656 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:45.148774 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:45.153162 (+ 4388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:45.153175 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:45.153192 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:45.153230 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:45.156459 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:45.157202 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:45.157209 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:45.157211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:45.157345 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:45.157712 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:45.157716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:45.157718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:45.158112 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:45.158412 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:45.158622 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 20:28:45.158850 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":379,"threads_started":1}
I20260929 20:28:45.160693 30154 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:28:45.246381 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 30031
I20260929 20:28:45.264040 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 29670
I20260929 20:28:45.265350 29732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:28:45.372476 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 29670
2026-09-29T20:28:45Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4138 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:28:45 dist-test-slave-p29l krb5kdc[30181](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:28:45 dist-test-slave-p29l krb5kdc[30181](info): set up 2 sockets
Sep 29 20:28:45 dist-test-slave-p29l krb5kdc[30181](info): commencing operation
krb5kdc: starting...
W20260929 20:28:46.459220 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.005s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:28:46 dist-test-slave-p29l krb5kdc[30181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:28:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:28:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.244.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
I20260929 20:28:46.647678 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:39467
--webserver_interface=127.28.244.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:44283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:39467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:28:46.979713 30194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:28:46.980060 30194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:28:46.980221 30194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:28:46.987705 30194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:28:46.987849 30194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:28:46.987900 30194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:28:46.988014 30194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:28:46.988097 30194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:28:46.998250 30194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:44283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:39467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:39467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:28:46.999754 30194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:28:47.001258 30194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:28:47.015028 30199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:47.015101 30200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:47.015149 30202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:28:47.016144 30194 server_base.cc:1034] running on GCE node
I20260929 20:28:47.017064 30194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:28:47.018581 30194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:28:47.019790 30194 hybrid_clock.cc:648] HybridClock initialized: now 1790713727019750 us; error 32 us; skew 500 ppm
I20260929 20:28:47.020210 30194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:28:47 dist-test-slave-p29l krb5kdc[30181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713727, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:28:47.027936 30194 init.cc:377] Logged in from keytab as kudu/127.28.244.190@KRBTEST.COM (short username kudu)
I20260929 20:28:47.029572 30194 webserver.cc:460] Webserver started at http://127.28.244.190:39443/ using document root <none> and password file <none>
I20260929 20:28:47.030346 30194 fs_manager.cc:359] Metadata directory not provided
I20260929 20:28:47.030515 30194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:28:47.030854 30194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:28:47.034092 30194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "1f822b7494584b879fc01720a2e5313b"
format_stamp: "Formatted at 2026-09-29 20:28:47 on dist-test-slave-p29l"
server_key: "b573aeee3e2d36cb871b16581ec1a6ff"
server_key_iv: "27a2af94d38273f1cbedc9fda917a0ae"
server_key_version: "encryptionkey@0"
I20260929 20:28:47.034916 30194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "1f822b7494584b879fc01720a2e5313b"
format_stamp: "Formatted at 2026-09-29 20:28:47 on dist-test-slave-p29l"
server_key: "b573aeee3e2d36cb871b16581ec1a6ff"
server_key_iv: "27a2af94d38273f1cbedc9fda917a0ae"
server_key_version: "encryptionkey@0"
I20260929 20:28:47.040531 30194 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 20:28:47.044291 30209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:47.045869 30194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:28:47.046039 30194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "1f822b7494584b879fc01720a2e5313b"
format_stamp: "Formatted at 2026-09-29 20:28:47 on dist-test-slave-p29l"
server_key: "b573aeee3e2d36cb871b16581ec1a6ff"
server_key_iv: "27a2af94d38273f1cbedc9fda917a0ae"
server_key_version: "encryptionkey@0"
I20260929 20:28:47.046231 30194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:28:47.071233 30194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:28:47.076294 30194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:28:47.076615 30194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:28:47.096161 30194 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:39467
I20260929 20:28:47.096160 30260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:39467 every 8 connection(s)
I20260929 20:28:47.097779 30194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:28:47.101967 30261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:28:47.106535 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 30194
I20260929 20:28:47.106781 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:28:47.107452 29650 external_mini_cluster.cc:1428] Setting key 9f5984c414071ce1ad313c7234eb8cd5
I20260929 20:28:47.115754 30261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f822b7494584b879fc01720a2e5313b: Bootstrap starting.
I20260929 20:28:47.120031 30261 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f822b7494584b879fc01720a2e5313b: Neither blocks nor log segments found. Creating new log.
I20260929 20:28:47.121533 30261 log.cc:824] T 00000000000000000000000000000000 P 1f822b7494584b879fc01720a2e5313b: Log is configured to *not* fsync() on all Append() calls
I20260929 20:28:47.125042 30261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f822b7494584b879fc01720a2e5313b: No bootstrap required, opened a new log
Sep 29 20:28:47 dist-test-slave-p29l krb5kdc[30181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:28:47.136298 30264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:47.109837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56396 (local address 127.28.244.190:39467)
0929 20:28:47.110844 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:47.110867 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:47.110938 (+ 71us) server_negotiation.cc:407] Connection header received
0929 20:28:47.113706 (+ 2768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:47.113748 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:47.114397 (+ 649us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:47.114929 (+ 532us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:47.116635 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:47.118811 (+ 2176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:47.120513 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:47.121723 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:47.121937 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:47.126266 (+ 4329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:47.126280 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:47.126303 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:47.126358 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:47.130739 (+ 4381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:47.131676 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:47.131686 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:47.131697 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:47.131992 (+ 295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:47.132418 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:47.132424 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:47.132427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:47.133265 (+ 838us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:47.133589 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:47.134144 (+ 555us) server_negotiation.cc:299] Negotiation successful
0929 20:28:47.134697 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":718,"thread_start_us":468,"threads_started":1}
I20260929 20:28:47.136345 30261 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f822b7494584b879fc01720a2e5313b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f822b7494584b879fc01720a2e5313b" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39467 } }
I20260929 20:28:47.136837 30261 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f822b7494584b879fc01720a2e5313b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:28:47.136960 30261 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f822b7494584b879fc01720a2e5313b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f822b7494584b879fc01720a2e5313b, State: Initialized, Role: FOLLOWER
I20260929 20:28:47.137702 30261 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f822b7494584b879fc01720a2e5313b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f822b7494584b879fc01720a2e5313b" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39467 } }
I20260929 20:28:47.138049 30261 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f822b7494584b879fc01720a2e5313b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:28:47.138206 30261 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f822b7494584b879fc01720a2e5313b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:28:47.138401 30261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f822b7494584b879fc01720a2e5313b [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:28:47.141121 30261 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f822b7494584b879fc01720a2e5313b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f822b7494584b879fc01720a2e5313b" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39467 } }
I20260929 20:28:47.141714 30261 leader_election.cc:302] T 00000000000000000000000000000000 P 1f822b7494584b879fc01720a2e5313b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f822b7494584b879fc01720a2e5313b; no voters:
I20260929 20:28:47.142472 30261 leader_election.cc:288] T 00000000000000000000000000000000 P 1f822b7494584b879fc01720a2e5313b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:28:47.142656 30266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f822b7494584b879fc01720a2e5313b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:28:47.143352 30266 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f822b7494584b879fc01720a2e5313b [term 1 LEADER]: Becoming Leader. State: Replica: 1f822b7494584b879fc01720a2e5313b, State: Running, Role: LEADER
I20260929 20:28:47.144148 30266 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f822b7494584b879fc01720a2e5313b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f822b7494584b879fc01720a2e5313b" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39467 } }
I20260929 20:28:47.144995 30261 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f822b7494584b879fc01720a2e5313b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:28:47.147636 30268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f822b7494584b879fc01720a2e5313b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f822b7494584b879fc01720a2e5313b. Latest consensus state: current_term: 1 leader_uuid: "1f822b7494584b879fc01720a2e5313b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f822b7494584b879fc01720a2e5313b" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39467 } } }
I20260929 20:28:47.148159 30268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f822b7494584b879fc01720a2e5313b [sys.catalog]: This master's current role is: LEADER
I20260929 20:28:47.147642 30267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f822b7494584b879fc01720a2e5313b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f822b7494584b879fc01720a2e5313b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f822b7494584b879fc01720a2e5313b" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39467 } } }
I20260929 20:28:47.148283 30267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f822b7494584b879fc01720a2e5313b [sys.catalog]: This master's current role is: LEADER
I20260929 20:28:47.150032 30272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:28:47.155897 30272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:28:47.166224 30272 catalog_manager.cc:1347] Generated new cluster ID: 91b14a69bd5047e7b81d935b6baea73d
I20260929 20:28:47.166363 30272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:28:47.179651 30272 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:28:47.181133 30272 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:28:47.193368 30272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f822b7494584b879fc01720a2e5313b: Generated new TSK 0
I20260929 20:28:47.194324 30272 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.244.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab.
Entry for principal kudu/127.28.244.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab.
I20260929 20:28:47.251627 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.129:0
--local_ip_for_outbound_sockets=127.28.244.129
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:39467
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:44283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:28:47.605726 30287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:28:47.606046 30287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:28:47.606216 30287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:28:47.613888 30287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:28:47.614028 30287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:28:47.614223 30287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.129
I20260929 20:28:47.624627 30287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:44283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:39467
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.129
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:28:47.626065 30287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:28:47.627463 30287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:28:47.641834 30292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:47.641783 30293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:47.641887 30295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:28:47.642797 30287 server_base.cc:1034] running on GCE node
I20260929 20:28:47.643411 30287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:28:47.644465 30287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:28:47.645670 30287 hybrid_clock.cc:648] HybridClock initialized: now 1790713727645593 us; error 64 us; skew 500 ppm
I20260929 20:28:47.646035 30287 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:28:47 dist-test-slave-p29l krb5kdc[30181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713727, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:28:47.649616 30287 init.cc:377] Logged in from keytab as kudu/127.28.244.129@KRBTEST.COM (short username kudu)
I20260929 20:28:47.651446 30287 webserver.cc:460] Webserver started at http://127.28.244.129:33965/ using document root <none> and password file <none>
I20260929 20:28:47.652278 30287 fs_manager.cc:359] Metadata directory not provided
I20260929 20:28:47.652439 30287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:28:47.652719 30287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:28:47.655761 30287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-0/data/instance:
uuid: "f0a6bd6f6f534477b9bcc837cf666a1e"
format_stamp: "Formatted at 2026-09-29 20:28:47 on dist-test-slave-p29l"
server_key: "7e7d46c2ab73d9a040f1a8d0cebbde69"
server_key_iv: "7a27e3382da141c8e6942574fb235ce0"
server_key_version: "encryptionkey@0"
I20260929 20:28:47.656641 30287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance:
uuid: "f0a6bd6f6f534477b9bcc837cf666a1e"
format_stamp: "Formatted at 2026-09-29 20:28:47 on dist-test-slave-p29l"
server_key: "7e7d46c2ab73d9a040f1a8d0cebbde69"
server_key_iv: "7a27e3382da141c8e6942574fb235ce0"
server_key_version: "encryptionkey@0"
I20260929 20:28:47.662247 30287 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:28:47.666109 30302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:47.667644 30287 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:28:47.667997 30287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-0/wal
uuid: "f0a6bd6f6f534477b9bcc837cf666a1e"
format_stamp: "Formatted at 2026-09-29 20:28:47 on dist-test-slave-p29l"
server_key: "7e7d46c2ab73d9a040f1a8d0cebbde69"
server_key_iv: "7a27e3382da141c8e6942574fb235ce0"
server_key_version: "encryptionkey@0"
I20260929 20:28:47.668219 30287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:28:47.686247 30287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:28:47.690944 30287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:28:47.691273 30287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:28:47.692508 30287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:28:47.694031 30287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:28:47.694123 30287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:47.694295 30287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:28:47.694351 30287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:47.727924 30287 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.129:40907
I20260929 20:28:47.728039 30414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.129:40907 every 8 connection(s)
I20260929 20:28:47.729313 30287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
I20260929 20:28:47.730958 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 30287
I20260929 20:28:47.731172 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance
I20260929 20:28:47.731873 29650 external_mini_cluster.cc:1428] Setting key 54576ce88159f38a6adb82fae491f443
Sep 29 20:28:47 dist-test-slave-p29l krb5kdc[30181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713727, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.244.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.130@KRBTEST.COM" created.
I20260929 20:28:47.751410 30417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:47.732145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:45507 (local address 127.28.244.190:39467)
0929 20:28:47.732680 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:47.732687 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:47.735784 (+ 3097us) server_negotiation.cc:407] Connection header received
0929 20:28:47.735910 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:47.735914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:47.735982 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:47.736101 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:47.737828 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:47.738555 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:47.740212 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:47.740617 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:47.740770 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:47.745050 (+ 4280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:47.745060 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:47.745069 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:47.745105 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:47.747981 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:47.748779 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:47.748785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:47.748789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:47.748922 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:47.749394 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:47.749399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:47.749404 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:47.749774 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:47.749947 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:47.750751 (+ 804us) server_negotiation.cc:299] Negotiation successful
0929 20:28:47.751021 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":262,"threads_started":1}
I20260929 20:28:47.752640 30418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:47.732828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39467 (local address 127.28.244.129:45507)
0929 20:28:47.733683 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:47.733730 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:47.735485 (+ 1755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:47.736364 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:47.736373 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:47.736896 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:28:47.737647 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:47.737668 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:47.738723 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:47.738728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:47.740063 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:47.740075 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:47.740868 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:47.740876 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:47.741137 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:47.741839 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:28:47.741867 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:28:47.744730 (+ 2863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:28:47.748148 (+ 3418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:47.748158 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:47.748178 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:47.748638 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:47.749044 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:47.749049 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:47.749053 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:47.749264 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:47.749905 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:28:47.749924 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:28:47.750340 (+ 416us) client_negotiation.cc:769] Sending connection context
0929 20:28:47.750627 (+ 287us) client_negotiation.cc:240] Negotiation successful
0929 20:28:47.751274 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":351,"threads_started":1}
I20260929 20:28:47.754311 30415 heartbeater.cc:347] Connected to a master server at 127.28.244.190:39467
I20260929 20:28:47.754676 30415 heartbeater.cc:464] Registering TS with master...
I20260929 20:28:47.755510 30415 heartbeater.cc:511] Master 127.28.244.190:39467 requested a full tablet report, sending...
I20260929 20:28:47.758054 30226 ts_manager.cc:194] Registered new tserver with Master: f0a6bd6f6f534477b9bcc837cf666a1e (127.28.244.129:40907)
I20260929 20:28:47.760314 30226 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.129@KRBTEST.COM'} at 127.28.244.129:45507
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab.
Entry for principal kudu/127.28.244.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab.
I20260929 20:28:47.774688 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.130:0
--local_ip_for_outbound_sockets=127.28.244.130
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:39467
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:44283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:28:48.113371 30421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:28:48.113708 30421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:28:48.113807 30421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:28:48.121503 30421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:28:48.121627 30421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:28:48.121747 30421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.130
I20260929 20:28:48.131898 30421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:44283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:39467
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.130
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:28:48.133247 30421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:28:48.134636 30421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:28:48.150686 30427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:48.151062 30429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:48.151182 30426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:28:48.152652 30421 server_base.cc:1034] running on GCE node
I20260929 20:28:48.153466 30421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:28:48.154471 30421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:28:48.155689 30421 hybrid_clock.cc:648] HybridClock initialized: now 1790713728155627 us; error 49 us; skew 500 ppm
I20260929 20:28:48.156106 30421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:28:48 dist-test-slave-p29l krb5kdc[30181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713728, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:28:48.159314 30421 init.cc:377] Logged in from keytab as kudu/127.28.244.130@KRBTEST.COM (short username kudu)
I20260929 20:28:48.160969 30421 webserver.cc:460] Webserver started at http://127.28.244.130:44341/ using document root <none> and password file <none>
I20260929 20:28:48.161741 30421 fs_manager.cc:359] Metadata directory not provided
I20260929 20:28:48.161893 30421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:28:48.162223 30421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:28:48.165351 30421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-1/data/instance:
uuid: "c76de86f63724b498dc23bce8848e895"
format_stamp: "Formatted at 2026-09-29 20:28:48 on dist-test-slave-p29l"
server_key: "7bd1ba128d97af65966d77744e6d743e"
server_key_iv: "833127e5b10d98b5b881cff840e3b894"
server_key_version: "encryptionkey@0"
I20260929 20:28:48.166118 30421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance:
uuid: "c76de86f63724b498dc23bce8848e895"
format_stamp: "Formatted at 2026-09-29 20:28:48 on dist-test-slave-p29l"
server_key: "7bd1ba128d97af65966d77744e6d743e"
server_key_iv: "833127e5b10d98b5b881cff840e3b894"
server_key_version: "encryptionkey@0"
I20260929 20:28:48.171669 30421 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 20:28:48.175583 30436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:48.177191 30421 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:28:48.177335 30421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-1/wal
uuid: "c76de86f63724b498dc23bce8848e895"
format_stamp: "Formatted at 2026-09-29 20:28:48 on dist-test-slave-p29l"
server_key: "7bd1ba128d97af65966d77744e6d743e"
server_key_iv: "833127e5b10d98b5b881cff840e3b894"
server_key_version: "encryptionkey@0"
I20260929 20:28:48.177520 30421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:28:48.195420 30421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:28:48.200102 30421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:28:48.200349 30421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:28:48.201287 30421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:28:48.202582 30421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:28:48.202656 30421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:48.202795 30421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:28:48.202847 30421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:48.236722 30421 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.130:45959
I20260929 20:28:48.236780 30548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.130:45959 every 8 connection(s)
I20260929 20:28:48.238109 30421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
I20260929 20:28:48.243716 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 30421
I20260929 20:28:48.243961 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance
I20260929 20:28:48.244410 29650 external_mini_cluster.cc:1428] Setting key 51fb9038a7bd854fbc475d5e64475e14
Sep 29 20:28:48 dist-test-slave-p29l krb5kdc[30181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713728, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:28:48.261505 30417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:48.240923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:46153 (local address 127.28.244.190:39467)
0929 20:28:48.241113 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:48.241117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:48.242375 (+ 1258us) server_negotiation.cc:407] Connection header received
0929 20:28:48.244478 (+ 2103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:48.244483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:48.244592 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:48.244706 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:48.246966 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:48.247682 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:48.249384 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:48.249832 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:48.249975 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:48.254523 (+ 4548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:48.254529 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:48.254532 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:48.254568 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:48.257285 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:48.258411 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:48.258416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:48.258418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:48.258569 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:48.259421 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:48.259425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:48.259428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:48.259734 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:48.259960 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:48.260978 (+ 1018us) server_negotiation.cc:299] Negotiation successful
0929 20:28:48.261185 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
WARNING: no policy specified for kudu/127.28.244.131@KRBTEST.COM; defaulting to no policy
I20260929 20:28:48.263048 30551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:48.241458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39467 (local address 127.28.244.130:46153)
0929 20:28:48.242193 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:48.242241 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:48.244168 (+ 1927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:48.245036 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:48.245049 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:48.245832 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:28:48.246784 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:48.246808 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:48.247879 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:48.247889 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:48.249219 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:48.249232 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:48.250101 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:48.250109 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:48.250350 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:48.251188 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:28:48.251221 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:28:48.254243 (+ 3022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:28:48.257461 (+ 3218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:48.257474 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:48.257507 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:48.258230 (+ 723us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:48.258909 (+ 679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:48.258918 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:48.258924 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:48.259253 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:48.259926 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:28:48.259935 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:28:48.260476 (+ 541us) client_negotiation.cc:769] Sending connection context
0929 20:28:48.260859 (+ 383us) client_negotiation.cc:240] Negotiation successful
0929 20:28:48.261524 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":287,"threads_started":1}
I20260929 20:28:48.264745 30549 heartbeater.cc:347] Connected to a master server at 127.28.244.190:39467
I20260929 20:28:48.265064 30549 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.131@KRBTEST.COM" created.
I20260929 20:28:48.265924 30549 heartbeater.cc:511] Master 127.28.244.190:39467 requested a full tablet report, sending...
I20260929 20:28:48.267590 30226 ts_manager.cc:194] Registered new tserver with Master: c76de86f63724b498dc23bce8848e895 (127.28.244.130:45959)
I20260929 20:28:48.268501 30226 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.130@KRBTEST.COM'} at 127.28.244.130:46153
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab.
Entry for principal kudu/127.28.244.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab.
I20260929 20:28:48.287530 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.131:0
--local_ip_for_outbound_sockets=127.28.244.131
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:39467
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:44283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:28:48.634961 30554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:28:48.635284 30554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:28:48.635382 30554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:28:48.643464 30554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:28:48.643625 30554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:28:48.643746 30554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.131
I20260929 20:28:48.654836 30554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:44283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:39467
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.131
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:28:48.656250 30554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:28:48.657616 30554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:28:48.671656 30559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:48.671924 30560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:48.671977 30562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:28:48.673085 30554 server_base.cc:1034] running on GCE node
I20260929 20:28:48.673831 30554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:28:48.674937 30554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:28:48.676227 30554 hybrid_clock.cc:648] HybridClock initialized: now 1790713728676174 us; error 56 us; skew 500 ppm
I20260929 20:28:48.676602 30554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:28:48 dist-test-slave-p29l krb5kdc[30181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713728, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:28:48.680078 30554 init.cc:377] Logged in from keytab as kudu/127.28.244.131@KRBTEST.COM (short username kudu)
I20260929 20:28:48.681836 30554 webserver.cc:460] Webserver started at http://127.28.244.131:43033/ using document root <none> and password file <none>
I20260929 20:28:48.682658 30554 fs_manager.cc:359] Metadata directory not provided
I20260929 20:28:48.682768 30554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:28:48.683058 30554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:28:48.686312 30554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-2/data/instance:
uuid: "a5b4765e41a64175a9bc92ecfde1e0e1"
format_stamp: "Formatted at 2026-09-29 20:28:48 on dist-test-slave-p29l"
server_key: "61bb9d00fa05a5fbc02f3b98d8f0130b"
server_key_iv: "1c8aa25db79c0ba3ebe40ececf228895"
server_key_version: "encryptionkey@0"
I20260929 20:28:48.687145 30554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance:
uuid: "a5b4765e41a64175a9bc92ecfde1e0e1"
format_stamp: "Formatted at 2026-09-29 20:28:48 on dist-test-slave-p29l"
server_key: "61bb9d00fa05a5fbc02f3b98d8f0130b"
server_key_iv: "1c8aa25db79c0ba3ebe40ececf228895"
server_key_version: "encryptionkey@0"
I20260929 20:28:48.692767 30554 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 20:28:48.696584 30569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:48.698052 30554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:28:48.698240 30554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-2/wal
uuid: "a5b4765e41a64175a9bc92ecfde1e0e1"
format_stamp: "Formatted at 2026-09-29 20:28:48 on dist-test-slave-p29l"
server_key: "61bb9d00fa05a5fbc02f3b98d8f0130b"
server_key_iv: "1c8aa25db79c0ba3ebe40ececf228895"
server_key_version: "encryptionkey@0"
I20260929 20:28:48.698457 30554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:28:48.714027 30554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:28:48.719069 30554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:28:48.719386 30554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:28:48.720507 30554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:28:48.721935 30554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:28:48.722044 30554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:48.722191 30554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:28:48.722265 30554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:48.757097 30554 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.131:45861
I20260929 20:28:48.757138 30681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.131:45861 every 8 connection(s)
I20260929 20:28:48.758507 30554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
I20260929 20:28:48.763599 30415 heartbeater.cc:503] Master 127.28.244.190:39467 was elected leader, sending a full tablet report...
I20260929 20:28:48.766953 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 30554
I20260929 20:28:48.767155 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance
I20260929 20:28:48.767573 29650 external_mini_cluster.cc:1428] Setting key 4b91b72ad02f8fd1ea0511b2f2da3921
Sep 29 20:28:48 dist-test-slave-p29l krb5kdc[30181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713728, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:28:48.779747 30417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:48.761311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:46113 (local address 127.28.244.190:39467)
0929 20:28:48.761490 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:48.761495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:48.762730 (+ 1235us) server_negotiation.cc:407] Connection header received
0929 20:28:48.764878 (+ 2148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:48.764883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:48.764978 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:48.765090 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:48.766728 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:48.767601 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:48.769497 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:48.770152 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:48.770286 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:48.774123 (+ 3837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:48.774129 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:48.774132 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:48.774168 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:48.776890 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:48.777663 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:48.777667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:48.777669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:48.777804 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:48.778187 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:48.778190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:48.778192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:48.778464 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:48.778588 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:48.779308 (+ 720us) server_negotiation.cc:299] Negotiation successful
0929 20:28:48.779492 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:28:48.780861 30684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:48.761862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39467 (local address 127.28.244.131:46113)
0929 20:28:48.762557 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:48.762596 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:48.764559 (+ 1963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:48.765337 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:48.765346 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:48.765879 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:28:48.766507 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:48.766526 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:48.767841 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:48.767850 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:48.769330 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:48.769344 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:48.770359 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:48.770387 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:48.770735 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:48.771448 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:28:48.771473 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:28:48.773888 (+ 2415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:28:48.777053 (+ 3165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:48.777061 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:48.777077 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:48.777541 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:48.777896 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:48.777900 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:48.777902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:48.778089 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:48.778582 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:28:48.778588 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:28:48.778956 (+ 368us) client_negotiation.cc:769] Sending connection context
0929 20:28:48.779197 (+ 241us) client_negotiation.cc:240] Negotiation successful
0929 20:28:48.779623 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":271,"threads_started":1}
I20260929 20:28:48.782534 30682 heartbeater.cc:347] Connected to a master server at 127.28.244.190:39467
I20260929 20:28:48.782891 30682 heartbeater.cc:464] Registering TS with master...
I20260929 20:28:48.783740 30682 heartbeater.cc:511] Master 127.28.244.190:39467 requested a full tablet report, sending...
I20260929 20:28:48.785233 30225 ts_manager.cc:194] Registered new tserver with Master: a5b4765e41a64175a9bc92ecfde1e0e1 (127.28.244.131:45861)
I20260929 20:28:48.786120 30225 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.131@KRBTEST.COM'} at 127.28.244.131:46113
I20260929 20:28:48.795948 29650 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:28:48.815652 30417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:48.802371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51462 (local address 127.28.244.190:39467)
0929 20:28:48.802595 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:48.802603 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:48.803062 (+ 459us) server_negotiation.cc:407] Connection header received
0929 20:28:48.803282 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:48.803287 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:48.803356 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:48.803472 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:48.805047 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:48.805804 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:48.807126 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:48.807582 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:48.807730 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:48.810100 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:48.810106 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:48.810109 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:48.810145 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:48.812854 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:48.813541 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:48.813548 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:48.813552 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:48.813694 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:48.814085 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:48.814092 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:48.814097 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:48.814449 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:48.814725 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:48.815086 (+ 361us) server_negotiation.cc:299] Negotiation successful
0929 20:28:48.815312 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:28:48.822811 30225 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51462:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:28:48.829900 30225 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:28:48.855340 30695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:48.843010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:40907 (local address 127.0.0.1:49288)
0929 20:28:48.843884 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:48.843937 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:48.844117 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:48.846160 (+ 2043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:48.846168 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:48.846201 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:28:48.846730 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:48.846742 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:48.849173 (+ 2431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:48.849178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:48.852132 (+ 2954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:48.852150 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:48.854160 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:48.854168 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:48.854459 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:48.854528 (+ 69us) client_negotiation.cc:769] Sending connection context
0929 20:28:48.854655 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 20:28:48.854860 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"mutex_wait_us":37,"thread_start_us":2095,"threads_started":1}
I20260929 20:28:48.855400 30697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:48.843647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54946 (local address 127.28.244.130:45959)
0929 20:28:48.844955 (+ 1308us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:48.844966 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:48.845519 (+ 553us) server_negotiation.cc:407] Connection header received
0929 20:28:48.845802 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:48.845810 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:48.846034 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:48.846203 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:28:48.847210 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:48.848982 (+ 1772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:48.852522 (+ 3540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:48.853851 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:48.854032 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:48.854252 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:48.854868 (+ 616us) server_negotiation.cc:299] Negotiation successful
0929 20:28:48.855058 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1130,"thread_start_us":1000,"threads_started":1}
I20260929 20:28:48.855695 30696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:48.843200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49288 (local address 127.28.244.129:40907)
0929 20:28:48.845387 (+ 2187us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:48.845396 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:48.845432 (+ 36us) server_negotiation.cc:407] Connection header received
0929 20:28:48.845549 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:48.845558 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:48.845847 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:48.847004 (+ 1157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:28:48.847296 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:48.848982 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:48.852366 (+ 3384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:48.853896 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:48.854793 (+ 897us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:48.854973 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:48.855038 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 20:28:48.855196 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2013,"thread_start_us":1864,"threads_started":1}
I20260929 20:28:48.856472 30694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:48.843157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:45959 (local address 127.0.0.1:54946)
0929 20:28:48.845395 (+ 2238us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:48.845414 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:48.845623 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:48.846523 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:48.846527 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:48.846560 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:28:48.846985 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:48.846997 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:48.849183 (+ 2186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:48.849187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:48.852355 (+ 3168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:48.852366 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:48.854431 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:48.854438 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:48.854614 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:48.854669 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 20:28:48.856050 (+ 1381us) client_negotiation.cc:240] Negotiation successful
0929 20:28:48.856161 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2100,"mutex_wait_us":7,"thread_start_us":664,"threads_started":1}
I20260929 20:28:48.859872 30698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:48.845531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:45861 (local address 127.0.0.1:33252)
0929 20:28:48.850243 (+ 4712us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:48.850266 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:48.850428 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:48.851565 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:48.851568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:48.851593 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:28:48.852081 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:48.852100 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:48.854245 (+ 2145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:48.854249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:48.856509 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:48.856529 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:48.859004 (+ 2475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:48.859011 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:48.859191 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:48.859225 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 20:28:48.859323 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 20:28:48.859477 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4569,"thread_start_us":4392,"threads_started":1}
I20260929 20:28:48.859980 30699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:48.845962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33252 (local address 127.28.244.131:45861)
0929 20:28:48.849778 (+ 3816us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:48.849788 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:48.850572 (+ 784us) server_negotiation.cc:407] Connection header received
0929 20:28:48.850692 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:48.850700 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:48.851052 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:48.851279 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:28:48.852373 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:48.854047 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:48.856722 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:48.858771 (+ 2049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:48.858974 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:48.859148 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:48.859436 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 20:28:48.859608 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3622,"thread_start_us":3399,"threads_started":1}
I20260929 20:28:48.860188 30350 tablet_service.cc:1467] Processing CreateTablet for tablet 4b347102618849d3ab809981dad0b650 (DEFAULT_TABLE table=test-table [id=0293a4f664f04ebf884cd3994e856140]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:28:48.860646 30484 tablet_service.cc:1467] Processing CreateTablet for tablet 4b347102618849d3ab809981dad0b650 (DEFAULT_TABLE table=test-table [id=0293a4f664f04ebf884cd3994e856140]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:28:48.861922 30350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b347102618849d3ab809981dad0b650. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:28:48.862021 30484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b347102618849d3ab809981dad0b650. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:28:48.864246 30617 tablet_service.cc:1467] Processing CreateTablet for tablet 4b347102618849d3ab809981dad0b650 (DEFAULT_TABLE table=test-table [id=0293a4f664f04ebf884cd3994e856140]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:28:48.866390 30617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b347102618849d3ab809981dad0b650. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:28:48.874761 30700 tablet_bootstrap.cc:492] T 4b347102618849d3ab809981dad0b650 P f0a6bd6f6f534477b9bcc837cf666a1e: Bootstrap starting.
I20260929 20:28:48.878697 30700 tablet_bootstrap.cc:654] T 4b347102618849d3ab809981dad0b650 P f0a6bd6f6f534477b9bcc837cf666a1e: Neither blocks nor log segments found. Creating new log.
I20260929 20:28:48.880227 30700 log.cc:824] T 4b347102618849d3ab809981dad0b650 P f0a6bd6f6f534477b9bcc837cf666a1e: Log is configured to *not* fsync() on all Append() calls
I20260929 20:28:48.881901 30701 tablet_bootstrap.cc:492] T 4b347102618849d3ab809981dad0b650 P c76de86f63724b498dc23bce8848e895: Bootstrap starting.
I20260929 20:28:48.884065 30700 tablet_bootstrap.cc:492] T 4b347102618849d3ab809981dad0b650 P f0a6bd6f6f534477b9bcc837cf666a1e: No bootstrap required, opened a new log
I20260929 20:28:48.884469 30700 ts_tablet_manager.cc:1397] T 4b347102618849d3ab809981dad0b650 P f0a6bd6f6f534477b9bcc837cf666a1e: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260929 20:28:48.885700 30703 tablet_bootstrap.cc:492] T 4b347102618849d3ab809981dad0b650 P a5b4765e41a64175a9bc92ecfde1e0e1: Bootstrap starting.
I20260929 20:28:48.885776 30701 tablet_bootstrap.cc:654] T 4b347102618849d3ab809981dad0b650 P c76de86f63724b498dc23bce8848e895: Neither blocks nor log segments found. Creating new log.
I20260929 20:28:48.887140 30701 log.cc:824] T 4b347102618849d3ab809981dad0b650 P c76de86f63724b498dc23bce8848e895: Log is configured to *not* fsync() on all Append() calls
I20260929 20:28:48.889123 30703 tablet_bootstrap.cc:654] T 4b347102618849d3ab809981dad0b650 P a5b4765e41a64175a9bc92ecfde1e0e1: Neither blocks nor log segments found. Creating new log.
I20260929 20:28:48.890153 30703 log.cc:824] T 4b347102618849d3ab809981dad0b650 P a5b4765e41a64175a9bc92ecfde1e0e1: Log is configured to *not* fsync() on all Append() calls
I20260929 20:28:48.890570 30701 tablet_bootstrap.cc:492] T 4b347102618849d3ab809981dad0b650 P c76de86f63724b498dc23bce8848e895: No bootstrap required, opened a new log
I20260929 20:28:48.890851 30701 ts_tablet_manager.cc:1397] T 4b347102618849d3ab809981dad0b650 P c76de86f63724b498dc23bce8848e895: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260929 20:28:48.893059 30703 tablet_bootstrap.cc:492] T 4b347102618849d3ab809981dad0b650 P a5b4765e41a64175a9bc92ecfde1e0e1: No bootstrap required, opened a new log
I20260929 20:28:48.893349 30703 ts_tablet_manager.cc:1397] T 4b347102618849d3ab809981dad0b650 P a5b4765e41a64175a9bc92ecfde1e0e1: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 20:28:48.895218 30700 raft_consensus.cc:348] T 4b347102618849d3ab809981dad0b650 P f0a6bd6f6f534477b9bcc837cf666a1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a6bd6f6f534477b9bcc837cf666a1e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 40907 } } peers { permanent_uuid: "c76de86f63724b498dc23bce8848e895" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45959 } } peers { permanent_uuid: "a5b4765e41a64175a9bc92ecfde1e0e1" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 45861 } }
I20260929 20:28:48.895673 30700 raft_consensus.cc:374] T 4b347102618849d3ab809981dad0b650 P f0a6bd6f6f534477b9bcc837cf666a1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:28:48.895773 30700 raft_consensus.cc:729] T 4b347102618849d3ab809981dad0b650 P f0a6bd6f6f534477b9bcc837cf666a1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a6bd6f6f534477b9bcc837cf666a1e, State: Initialized, Role: FOLLOWER
I20260929 20:28:48.896473 30700 consensus_queue.cc:260] T 4b347102618849d3ab809981dad0b650 P f0a6bd6f6f534477b9bcc837cf666a1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a6bd6f6f534477b9bcc837cf666a1e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 40907 } } peers { permanent_uuid: "c76de86f63724b498dc23bce8848e895" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45959 } } peers { permanent_uuid: "a5b4765e41a64175a9bc92ecfde1e0e1" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 45861 } }
I20260929 20:28:48.898443 30700 ts_tablet_manager.cc:1428] T 4b347102618849d3ab809981dad0b650 P f0a6bd6f6f534477b9bcc837cf666a1e: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260929 20:28:48.901291 30701 raft_consensus.cc:348] T 4b347102618849d3ab809981dad0b650 P c76de86f63724b498dc23bce8848e895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a6bd6f6f534477b9bcc837cf666a1e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 40907 } } peers { permanent_uuid: "c76de86f63724b498dc23bce8848e895" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45959 } } peers { permanent_uuid: "a5b4765e41a64175a9bc92ecfde1e0e1" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 45861 } }
I20260929 20:28:48.901783 30701 raft_consensus.cc:374] T 4b347102618849d3ab809981dad0b650 P c76de86f63724b498dc23bce8848e895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:28:48.901895 30701 raft_consensus.cc:729] T 4b347102618849d3ab809981dad0b650 P c76de86f63724b498dc23bce8848e895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c76de86f63724b498dc23bce8848e895, State: Initialized, Role: FOLLOWER
I20260929 20:28:48.901722 30703 raft_consensus.cc:348] T 4b347102618849d3ab809981dad0b650 P a5b4765e41a64175a9bc92ecfde1e0e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a6bd6f6f534477b9bcc837cf666a1e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 40907 } } peers { permanent_uuid: "c76de86f63724b498dc23bce8848e895" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45959 } } peers { permanent_uuid: "a5b4765e41a64175a9bc92ecfde1e0e1" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 45861 } }
I20260929 20:28:48.902254 30703 raft_consensus.cc:374] T 4b347102618849d3ab809981dad0b650 P a5b4765e41a64175a9bc92ecfde1e0e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:28:48.902387 30703 raft_consensus.cc:729] T 4b347102618849d3ab809981dad0b650 P a5b4765e41a64175a9bc92ecfde1e0e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b4765e41a64175a9bc92ecfde1e0e1, State: Initialized, Role: FOLLOWER
I20260929 20:28:48.902678 30701 consensus_queue.cc:260] T 4b347102618849d3ab809981dad0b650 P c76de86f63724b498dc23bce8848e895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a6bd6f6f534477b9bcc837cf666a1e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 40907 } } peers { permanent_uuid: "c76de86f63724b498dc23bce8848e895" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45959 } } peers { permanent_uuid: "a5b4765e41a64175a9bc92ecfde1e0e1" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 45861 } }
I20260929 20:28:48.903056 30703 consensus_queue.cc:260] T 4b347102618849d3ab809981dad0b650 P a5b4765e41a64175a9bc92ecfde1e0e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a6bd6f6f534477b9bcc837cf666a1e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 40907 } } peers { permanent_uuid: "c76de86f63724b498dc23bce8848e895" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45959 } } peers { permanent_uuid: "a5b4765e41a64175a9bc92ecfde1e0e1" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 45861 } }
I20260929 20:28:48.905061 30701 ts_tablet_manager.cc:1428] T 4b347102618849d3ab809981dad0b650 P c76de86f63724b498dc23bce8848e895: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260929 20:28:48.905520 30549 heartbeater.cc:503] Master 127.28.244.190:39467 was elected leader, sending a full tablet report...
I20260929 20:28:48.905949 30682 heartbeater.cc:503] Master 127.28.244.190:39467 was elected leader, sending a full tablet report...
I20260929 20:28:48.906649 30703 ts_tablet_manager.cc:1428] T 4b347102618849d3ab809981dad0b650 P a5b4765e41a64175a9bc92ecfde1e0e1: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260929 20:28:48.943040 30706 raft_consensus.cc:482] T 4b347102618849d3ab809981dad0b650 P f0a6bd6f6f534477b9bcc837cf666a1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:28:48.943337 30706 raft_consensus.cc:504] T 4b347102618849d3ab809981dad0b650 P f0a6bd6f6f534477b9bcc837cf666a1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a6bd6f6f534477b9bcc837cf666a1e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 40907 } } peers { permanent_uuid: "c76de86f63724b498dc23bce8848e895" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45959 } } peers { permanent_uuid: "a5b4765e41a64175a9bc92ecfde1e0e1" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 45861 } }
I20260929 20:28:48.945286 30706 leader_election.cc:288] T 4b347102618849d3ab809981dad0b650 P f0a6bd6f6f534477b9bcc837cf666a1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c76de86f63724b498dc23bce8848e895 (127.28.244.130:45959), a5b4765e41a64175a9bc92ecfde1e0e1 (127.28.244.131:45861)
I20260929 20:28:48.954931 30699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:48.946024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:46681 (local address 127.28.244.131:45861)
0929 20:28:48.946314 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:48.946323 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:48.946963 (+ 640us) server_negotiation.cc:407] Connection header received
0929 20:28:48.947118 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:48.947124 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:48.947203 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:48.947309 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:28:48.948199 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:48.949350 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:48.952786 (+ 3436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:48.954045 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:48.954213 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:48.954289 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:48.954635 (+ 346us) server_negotiation.cc:299] Negotiation successful
0929 20:28:48.954742 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 20:28:48.955056 30710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:48.945745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:45861 (local address 127.28.244.129:46681)
0929 20:28:48.946858 (+ 1113us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:48.946879 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:48.946972 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:48.947348 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:48.947353 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:48.947395 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:28:48.948038 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:48.948047 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:48.949540 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:48.949547 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:48.952587 (+ 3040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:48.952608 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:48.954326 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:48.954333 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:48.954475 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:48.954504 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 20:28:48.954594 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 20:28:48.954701 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":995,"mutex_wait_us":142,"spinlock_wait_cycles":6912,"thread_start_us":516,"threads_started":1}
I20260929 20:28:48.956176 30697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:48.945716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:59683 (local address 127.28.244.130:45959)
0929 20:28:48.945970 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:48.945978 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:48.946881 (+ 903us) server_negotiation.cc:407] Connection header received
0929 20:28:48.947118 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:48.947124 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:48.947203 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:48.947309 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:28:48.948199 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:48.949350 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:48.952784 (+ 3434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:48.954044 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:48.954212 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:48.954289 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:48.955712 (+ 1423us) server_negotiation.cc:299] Negotiation successful
0929 20:28:48.955904 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:28:48.956243 30709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:48.945741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:45959 (local address 127.28.244.129:59683)
0929 20:28:48.946760 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:48.946782 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:48.946969 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:48.947348 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:48.947353 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:48.947388 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:28:48.948026 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:48.948041 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:48.949541 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:48.949547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:48.952608 (+ 3061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:48.952618 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:48.955381 (+ 2763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:48.955388 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:48.955527 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:48.955552 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 20:28:48.955664 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 20:28:48.955805 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":863,"mutex_wait_us":214,"spinlock_wait_cycles":896,"thread_start_us":561,"threads_started":1}
I20260929 20:28:48.956183 30637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b347102618849d3ab809981dad0b650" candidate_uuid: "f0a6bd6f6f534477b9bcc837cf666a1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b4765e41a64175a9bc92ecfde1e0e1" is_pre_election: true
I20260929 20:28:48.956861 30637 raft_consensus.cc:2454] T 4b347102618849d3ab809981dad0b650 P a5b4765e41a64175a9bc92ecfde1e0e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0a6bd6f6f534477b9bcc837cf666a1e in term 0.
I20260929 20:28:48.957603 30303 leader_election.cc:302] T 4b347102618849d3ab809981dad0b650 P f0a6bd6f6f534477b9bcc837cf666a1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5b4765e41a64175a9bc92ecfde1e0e1, f0a6bd6f6f534477b9bcc837cf666a1e; no voters:
I20260929 20:28:48.958067 30706 raft_consensus.cc:2781] T 4b347102618849d3ab809981dad0b650 P f0a6bd6f6f534477b9bcc837cf666a1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:28:48.958240 30706 raft_consensus.cc:482] T 4b347102618849d3ab809981dad0b650 P f0a6bd6f6f534477b9bcc837cf666a1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:28:48.958117 30504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b347102618849d3ab809981dad0b650" candidate_uuid: "f0a6bd6f6f534477b9bcc837cf666a1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c76de86f63724b498dc23bce8848e895" is_pre_election: true
I20260929 20:28:48.958355 30706 raft_consensus.cc:3037] T 4b347102618849d3ab809981dad0b650 P f0a6bd6f6f534477b9bcc837cf666a1e [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:28:48.958691 30504 raft_consensus.cc:2454] T 4b347102618849d3ab809981dad0b650 P c76de86f63724b498dc23bce8848e895 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0a6bd6f6f534477b9bcc837cf666a1e in term 0.
I20260929 20:28:48.961108 30706 raft_consensus.cc:504] T 4b347102618849d3ab809981dad0b650 P f0a6bd6f6f534477b9bcc837cf666a1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a6bd6f6f534477b9bcc837cf666a1e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 40907 } } peers { permanent_uuid: "c76de86f63724b498dc23bce8848e895" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45959 } } peers { permanent_uuid: "a5b4765e41a64175a9bc92ecfde1e0e1" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 45861 } }
I20260929 20:28:48.961769 30706 leader_election.cc:288] T 4b347102618849d3ab809981dad0b650 P f0a6bd6f6f534477b9bcc837cf666a1e [CANDIDATE]: Term 1 election: Requested vote from peers c76de86f63724b498dc23bce8848e895 (127.28.244.130:45959), a5b4765e41a64175a9bc92ecfde1e0e1 (127.28.244.131:45861)
I20260929 20:28:48.962234 30504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b347102618849d3ab809981dad0b650" candidate_uuid: "f0a6bd6f6f534477b9bcc837cf666a1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c76de86f63724b498dc23bce8848e895"
I20260929 20:28:48.962363 30637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b347102618849d3ab809981dad0b650" candidate_uuid: "f0a6bd6f6f534477b9bcc837cf666a1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b4765e41a64175a9bc92ecfde1e0e1"
I20260929 20:28:48.962443 30504 raft_consensus.cc:3037] T 4b347102618849d3ab809981dad0b650 P c76de86f63724b498dc23bce8848e895 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:28:48.962602 30637 raft_consensus.cc:3037] T 4b347102618849d3ab809981dad0b650 P a5b4765e41a64175a9bc92ecfde1e0e1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:28:48.964888 30504 raft_consensus.cc:2454] T 4b347102618849d3ab809981dad0b650 P c76de86f63724b498dc23bce8848e895 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0a6bd6f6f534477b9bcc837cf666a1e in term 1.
I20260929 20:28:48.964903 30637 raft_consensus.cc:2454] T 4b347102618849d3ab809981dad0b650 P a5b4765e41a64175a9bc92ecfde1e0e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0a6bd6f6f534477b9bcc837cf666a1e in term 1.
I20260929 20:28:48.965345 30303 leader_election.cc:302] T 4b347102618849d3ab809981dad0b650 P f0a6bd6f6f534477b9bcc837cf666a1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5b4765e41a64175a9bc92ecfde1e0e1, f0a6bd6f6f534477b9bcc837cf666a1e; no voters:
I20260929 20:28:48.965696 30706 raft_consensus.cc:2781] T 4b347102618849d3ab809981dad0b650 P f0a6bd6f6f534477b9bcc837cf666a1e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:28:48.966189 30706 raft_consensus.cc:686] T 4b347102618849d3ab809981dad0b650 P f0a6bd6f6f534477b9bcc837cf666a1e [term 1 LEADER]: Becoming Leader. State: Replica: f0a6bd6f6f534477b9bcc837cf666a1e, State: Running, Role: LEADER
I20260929 20:28:48.966687 30706 consensus_queue.cc:237] T 4b347102618849d3ab809981dad0b650 P f0a6bd6f6f534477b9bcc837cf666a1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a6bd6f6f534477b9bcc837cf666a1e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 40907 } } peers { permanent_uuid: "c76de86f63724b498dc23bce8848e895" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45959 } } peers { permanent_uuid: "a5b4765e41a64175a9bc92ecfde1e0e1" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 45861 } }
I20260929 20:28:48.971555 30225 catalog_manager.cc:5495] T 4b347102618849d3ab809981dad0b650 P f0a6bd6f6f534477b9bcc837cf666a1e reported cstate change: term changed from 0 to 1, leader changed from <none> to f0a6bd6f6f534477b9bcc837cf666a1e (127.28.244.129). New cstate: current_term: 1 leader_uuid: "f0a6bd6f6f534477b9bcc837cf666a1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a6bd6f6f534477b9bcc837cf666a1e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 40907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c76de86f63724b498dc23bce8848e895" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5b4765e41a64175a9bc92ecfde1e0e1" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 45861 } health_report { overall_health: UNKNOWN } } }
W20260929 20:28:48.981881 30416 tablet.cc:2369] T 4b347102618849d3ab809981dad0b650 P f0a6bd6f6f534477b9bcc837cf666a1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 20:28:48 dist-test-slave-p29l krb5kdc[30181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.129@KRBTEST.COM
W20260929 20:28:48.990244 30550 tablet.cc:2369] T 4b347102618849d3ab809981dad0b650 P c76de86f63724b498dc23bce8848e895: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:28:48.996002 30696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:48.980393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49296 (local address 127.28.244.129:40907)
0929 20:28:48.980582 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:48.980587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:48.981162 (+ 575us) server_negotiation.cc:407] Connection header received
0929 20:28:48.981380 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:48.981384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:48.981471 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:48.981578 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:48.983290 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:48.984117 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:48.985400 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:48.985820 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:48.985957 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:48.990014 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:48.990030 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:48.990047 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:48.990084 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:48.993099 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:48.993827 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:48.993836 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:48.993839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:48.994004 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:48.994377 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:48.994383 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:48.994386 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:48.994749 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:48.995064 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:48.995366 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 20:28:48.995620 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260929 20:28:49.010263 30683 tablet.cc:2369] T 4b347102618849d3ab809981dad0b650 P a5b4765e41a64175a9bc92ecfde1e0e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 20:28:49 dist-test-slave-p29l krb5kdc[30181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713729, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 20:28:49 dist-test-slave-p29l krb5kdc[30181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713729, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.244.129@KRBTEST.COM
I20260929 20:28:49.044083 30696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:49.029458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49302 (local address 127.28.244.129:40907)
0929 20:28:49.029700 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:49.029707 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:49.030081 (+ 374us) server_negotiation.cc:407] Connection header received
0929 20:28:49.030308 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:49.030312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:49.030380 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:49.030513 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:49.032018 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:49.032774 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:49.034119 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:49.034570 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:49.034717 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:49.038918 (+ 4201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:49.038923 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:49.038926 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:49.038963 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:49.041739 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:49.042379 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:49.042383 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:49.042385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:49.042528 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:49.042868 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:49.042871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:49.042873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:49.043142 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:49.043349 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:49.043582 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 20:28:49.043759 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Sep 29 20:28:49 dist-test-slave-p29l krb5kdc[30181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 20:28:49 dist-test-slave-p29l krb5kdc[30181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.129@KRBTEST.COM
I20260929 20:28:49.090550 30696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:49.076261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49318 (local address 127.28.244.129:40907)
0929 20:28:49.076514 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:49.076523 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:49.076864 (+ 341us) server_negotiation.cc:407] Connection header received
0929 20:28:49.077095 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:49.077106 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:49.077186 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:49.077299 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:49.078711 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:49.079452 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:49.080790 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:49.081275 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:49.081415 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:49.085411 (+ 3996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:49.085416 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:49.085439 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:49.085475 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:49.088249 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:49.088892 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:49.088895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:49.088897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:49.089026 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:49.089361 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:49.089364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:49.089366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:49.089656 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:49.089852 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:49.090090 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 20:28:49.090268 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260929 20:28:49.092468 30330 tablet_service.cc:2908] Rejecting scan request for tablet 4b347102618849d3ab809981dad0b650: Uninitialized: clean time has not yet been initialized
I20260929 20:28:49.095640 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 30287
I20260929 20:28:49.113824 30696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:49.096908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49330 (local address 127.28.244.129:40907)
0929 20:28:49.097084 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:49.097089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:49.100310 (+ 3221us) server_negotiation.cc:407] Connection header received
0929 20:28:49.100437 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:49.100443 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:49.100544 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:49.100668 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:49.101984 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:49.103067 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:49.104419 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:49.105069 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:49.105196 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:49.107436 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:49.107446 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:49.107453 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:49.107504 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:49.110966 (+ 3462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:49.111693 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:49.111699 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:49.111703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:49.111921 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:49.112298 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:49.112303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:49.112307 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:49.112669 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:49.112955 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:49.113183 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 20:28:49.113420 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:28:49.114648 30410 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:28:49.209483 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 30287
I20260929 20:28:49.228440 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 30421
Sep 29 20:28:49 dist-test-slave-p29l krb5kdc[30181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.130@KRBTEST.COM
I20260929 20:28:49.244010 30697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:49.229285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54954 (local address 127.28.244.130:45959)
0929 20:28:49.229527 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:49.229532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:49.229550 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:28:49.229609 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:49.229612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:49.229700 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:49.229787 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:49.230968 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:49.231724 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:49.233043 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:49.233442 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:49.233541 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:49.237683 (+ 4142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:49.237695 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:49.237712 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:49.237752 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:49.240949 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:49.241721 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:49.241728 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:49.241730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:49.241869 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:49.242264 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:49.242270 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:49.242273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:49.242687 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:49.243028 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:49.243338 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 20:28:49.243574 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260929 20:28:49.244689 30544 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:28:49.335636 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 30421
I20260929 20:28:49.353829 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 30554
Sep 29 20:28:49 dist-test-slave-p29l krb5kdc[30181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.131@KRBTEST.COM
I20260929 20:28:49.369695 30699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:49.354669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33266 (local address 127.28.244.131:45861)
0929 20:28:49.354864 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:49.354871 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:49.354893 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:28:49.355119 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:49.355124 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:49.355193 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:49.355305 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:49.356691 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:49.357449 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:49.358947 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:49.359376 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:49.359531 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:49.363890 (+ 4359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:49.363908 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:49.363925 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:49.363969 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:49.366734 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:49.367442 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:49.367450 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:49.367452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:49.367611 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:49.368268 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:49.368272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:49.368274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:49.368659 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:49.368900 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:49.369248 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 20:28:49.369423 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:28:49.370402 30677 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:28:49.460495 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 30554
I20260929 20:28:49.477607 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 30194
I20260929 20:28:49.478973 30256 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:28:49.588271 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 30194
2026-09-29T20:28:49Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4214 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:28:49 dist-test-slave-p29l krb5kdc[30730](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:28:49 dist-test-slave-p29l krb5kdc[30730](info): set up 2 sockets
Sep 29 20:28:49 dist-test-slave-p29l krb5kdc[30730](info): commencing operation
krb5kdc: starting...
W20260929 20:28:50.751526 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.004s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:28:50 dist-test-slave-p29l krb5kdc[30730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:28:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:28:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.244.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
I20260929 20:28:50.948453 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:34953
--webserver_interface=127.28.244.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:33251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:34953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:28:51.296958 30743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:28:51.297482 30743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:28:51.297658 30743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:28:51.311977 30743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:28:51.312170 30743 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 20:28:51.312239 30743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:28:51.312433 30743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:28:51.312518 30743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:28:51.312638 30743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:28:51.332063 30743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:33251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:34953
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:34953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:28:51.333952 30743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:28:51.335948 30743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:28:51.352627 30751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:51.352504 30749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:51.353147 30748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:28:51.354590 30743 server_base.cc:1034] running on GCE node
I20260929 20:28:51.355641 30743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:28:51.357666 30743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:28:51.359018 30743 hybrid_clock.cc:648] HybridClock initialized: now 1790713731358916 us; error 65 us; skew 500 ppm
I20260929 20:28:51.359481 30743 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:28:51 dist-test-slave-p29l krb5kdc[30730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713731, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:28:51.363343 30743 init.cc:377] Logged in from keytab as kudu/127.28.244.190@KRBTEST.COM (short username kudu)
I20260929 20:28:51.365324 30743 webserver.cc:460] Webserver started at http://127.28.244.190:40955/ using document root <none> and password file <none>
I20260929 20:28:51.366220 30743 fs_manager.cc:359] Metadata directory not provided
I20260929 20:28:51.366369 30743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:28:51.366809 30743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:28:51.370265 30743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "0a039d1b8e4f4eb9849b1bf235f81620"
format_stamp: "Formatted at 2026-09-29 20:28:51 on dist-test-slave-p29l"
server_key: "282b8094ab10b267f7a0d7674883d727"
server_key_iv: "0ca0ee0505b162460621811acb4d3544"
server_key_version: "encryptionkey@0"
I20260929 20:28:51.371188 30743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "0a039d1b8e4f4eb9849b1bf235f81620"
format_stamp: "Formatted at 2026-09-29 20:28:51 on dist-test-slave-p29l"
server_key: "282b8094ab10b267f7a0d7674883d727"
server_key_iv: "0ca0ee0505b162460621811acb4d3544"
server_key_version: "encryptionkey@0"
I20260929 20:28:51.378201 30743 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260929 20:28:51.382378 30758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:51.384182 30743 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:28:51.384361 30743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "0a039d1b8e4f4eb9849b1bf235f81620"
format_stamp: "Formatted at 2026-09-29 20:28:51 on dist-test-slave-p29l"
server_key: "282b8094ab10b267f7a0d7674883d727"
server_key_iv: "0ca0ee0505b162460621811acb4d3544"
server_key_version: "encryptionkey@0"
I20260929 20:28:51.384608 30743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:28:51.405661 30743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:28:51.410485 30743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:28:51.410732 30743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:28:51.434053 30743 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:34953
I20260929 20:28:51.434079 30819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:34953 every 8 connection(s)
I20260929 20:28:51.435612 30743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:28:51.438722 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 30743
I20260929 20:28:51.439000 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:28:51.439869 29650 external_mini_cluster.cc:1428] Setting key 0201aabe813a984ddd8afd4d62a9fd0d
I20260929 20:28:51.440605 30820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:28:51.453869 30820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a039d1b8e4f4eb9849b1bf235f81620: Bootstrap starting.
Sep 29 20:28:51 dist-test-slave-p29l krb5kdc[30730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:28:51.457830 30820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a039d1b8e4f4eb9849b1bf235f81620: Neither blocks nor log segments found. Creating new log.
I20260929 20:28:51.459251 30820 log.cc:824] T 00000000000000000000000000000000 P 0a039d1b8e4f4eb9849b1bf235f81620: Log is configured to *not* fsync() on all Append() calls
I20260929 20:28:51.462565 30820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a039d1b8e4f4eb9849b1bf235f81620: No bootstrap required, opened a new log
I20260929 20:28:51.465793 30823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:51.442160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55852 (local address 127.28.244.190:34953)
0929 20:28:51.443026 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:51.443039 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:51.443084 (+ 45us) server_negotiation.cc:407] Connection header received
0929 20:28:51.445710 (+ 2626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:51.445737 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:51.446347 (+ 610us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:51.446833 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:51.448388 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:51.449743 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:51.451040 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:51.451873 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:51.452086 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:51.456974 (+ 4888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:51.456983 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:51.457007 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:51.457053 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:51.460798 (+ 3745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:51.461681 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:51.461689 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:51.461696 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:51.461912 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:51.462380 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:51.462384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:51.462387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:51.463110 (+ 723us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:51.463505 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:51.464077 (+ 572us) server_negotiation.cc:299] Negotiation successful
0929 20:28:51.464621 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":397,"threads_started":1}
I20260929 20:28:51.471297 30820 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a039d1b8e4f4eb9849b1bf235f81620 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a039d1b8e4f4eb9849b1bf235f81620" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34953 } }
I20260929 20:28:51.471720 30820 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a039d1b8e4f4eb9849b1bf235f81620 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:28:51.471850 30820 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a039d1b8e4f4eb9849b1bf235f81620 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a039d1b8e4f4eb9849b1bf235f81620, State: Initialized, Role: FOLLOWER
I20260929 20:28:51.472491 30820 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a039d1b8e4f4eb9849b1bf235f81620 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a039d1b8e4f4eb9849b1bf235f81620" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34953 } }
I20260929 20:28:51.472759 30820 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a039d1b8e4f4eb9849b1bf235f81620 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:28:51.472910 30820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a039d1b8e4f4eb9849b1bf235f81620 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:28:51.473116 30820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a039d1b8e4f4eb9849b1bf235f81620 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:28:51.475327 30820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a039d1b8e4f4eb9849b1bf235f81620 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a039d1b8e4f4eb9849b1bf235f81620" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34953 } }
I20260929 20:28:51.476022 30820 leader_election.cc:302] T 00000000000000000000000000000000 P 0a039d1b8e4f4eb9849b1bf235f81620 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a039d1b8e4f4eb9849b1bf235f81620; no voters:
I20260929 20:28:51.476764 30820 leader_election.cc:288] T 00000000000000000000000000000000 P 0a039d1b8e4f4eb9849b1bf235f81620 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:28:51.476894 30825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a039d1b8e4f4eb9849b1bf235f81620 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:28:51.477759 30825 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a039d1b8e4f4eb9849b1bf235f81620 [term 1 LEADER]: Becoming Leader. State: Replica: 0a039d1b8e4f4eb9849b1bf235f81620, State: Running, Role: LEADER
I20260929 20:28:51.478369 30825 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a039d1b8e4f4eb9849b1bf235f81620 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a039d1b8e4f4eb9849b1bf235f81620" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34953 } }
I20260929 20:28:51.479149 30820 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a039d1b8e4f4eb9849b1bf235f81620 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:28:51.481482 30827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a039d1b8e4f4eb9849b1bf235f81620 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a039d1b8e4f4eb9849b1bf235f81620. Latest consensus state: current_term: 1 leader_uuid: "0a039d1b8e4f4eb9849b1bf235f81620" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a039d1b8e4f4eb9849b1bf235f81620" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34953 } } }
I20260929 20:28:51.482007 30827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a039d1b8e4f4eb9849b1bf235f81620 [sys.catalog]: This master's current role is: LEADER
I20260929 20:28:51.481943 30826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a039d1b8e4f4eb9849b1bf235f81620 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a039d1b8e4f4eb9849b1bf235f81620" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a039d1b8e4f4eb9849b1bf235f81620" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34953 } } }
I20260929 20:28:51.482242 30826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a039d1b8e4f4eb9849b1bf235f81620 [sys.catalog]: This master's current role is: LEADER
I20260929 20:28:51.484516 30833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:28:51.490561 30833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:28:51.501747 30833 catalog_manager.cc:1347] Generated new cluster ID: 453cac45c83943aaa579af84c17f3c77
I20260929 20:28:51.501888 30833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:28:51.518214 30833 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:28:51.520247 30833 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:28:51.529219 30833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a039d1b8e4f4eb9849b1bf235f81620: Generated new TSK 0
I20260929 20:28:51.530364 30833 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.244.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab.
Entry for principal kudu/127.28.244.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab.
I20260929 20:28:51.592875 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.129:0
--local_ip_for_outbound_sockets=127.28.244.129
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:34953
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:33251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:28:51.953498 30846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:28:51.953809 30846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:28:51.953908 30846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:28:51.961589 30846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:28:51.961722 30846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:28:51.961809 30846 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:28:51.961898 30846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.129
I20260929 20:28:51.973032 30846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:33251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.244.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.244.190:34953
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.129
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:28:51.974395 30846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:28:51.975800 30846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:28:51.990893 30854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:51.990828 30852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:51.991142 30851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:28:51.991792 30846 server_base.cc:1034] running on GCE node
I20260929 20:28:51.992588 30846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:28:51.993691 30846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:28:51.994922 30846 hybrid_clock.cc:648] HybridClock initialized: now 1790713731994853 us; error 43 us; skew 500 ppm
I20260929 20:28:51.995261 30846 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:28:51 dist-test-slave-p29l krb5kdc[30730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713731, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:28:51.998905 30846 init.cc:377] Logged in from keytab as kudu/127.28.244.129@KRBTEST.COM (short username kudu)
I20260929 20:28:52.000780 30846 webserver.cc:460] Webserver started at http://127.28.244.129:37337/ using document root <none> and password file <none>
I20260929 20:28:52.001559 30846 fs_manager.cc:359] Metadata directory not provided
I20260929 20:28:52.001695 30846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:28:52.002048 30846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:28:52.005425 30846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-0/data/instance:
uuid: "ee58c80790c7457cbae8da3c197e1a8b"
format_stamp: "Formatted at 2026-09-29 20:28:51 on dist-test-slave-p29l"
server_key: "3424e2fc23746c1afa410a06090143bf"
server_key_iv: "e11f4ccd8ab0d82e05567b9750057aaa"
server_key_version: "encryptionkey@0"
I20260929 20:28:52.006341 30846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance:
uuid: "ee58c80790c7457cbae8da3c197e1a8b"
format_stamp: "Formatted at 2026-09-29 20:28:51 on dist-test-slave-p29l"
server_key: "3424e2fc23746c1afa410a06090143bf"
server_key_iv: "e11f4ccd8ab0d82e05567b9750057aaa"
server_key_version: "encryptionkey@0"
I20260929 20:28:52.011996 30846 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 20:28:52.015710 30861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:52.017427 30846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:28:52.017640 30846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-0/wal
uuid: "ee58c80790c7457cbae8da3c197e1a8b"
format_stamp: "Formatted at 2026-09-29 20:28:51 on dist-test-slave-p29l"
server_key: "3424e2fc23746c1afa410a06090143bf"
server_key_iv: "e11f4ccd8ab0d82e05567b9750057aaa"
server_key_version: "encryptionkey@0"
I20260929 20:28:52.017884 30846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:28:52.040802 30846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:28:52.045745 30846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:28:52.046046 30846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:28:52.049151 30846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:28:52.049263 30846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:52.049417 30846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:28:52.049515 30846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:28:52 dist-test-slave-p29l krb5kdc[30730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713731, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:28:52.080861 30870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:52.049947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:48305 (local address 127.28.244.190:34953)
0929 20:28:52.050556 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:52.050563 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:52.052136 (+ 1573us) server_negotiation.cc:407] Connection header received
0929 20:28:52.055457 (+ 3321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:52.055464 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:52.055614 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:52.055765 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:52.058631 (+ 2866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:52.059943 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:52.063586 (+ 3643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:52.064208 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:52.064339 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:52.070723 (+ 6384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:52.070735 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:52.070741 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:52.070802 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:52.075400 (+ 4598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:52.076854 (+ 1454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:52.076859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:52.076862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:52.077059 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:52.077858 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:52.077867 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:52.077873 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:52.078344 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:52.078530 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:52.079946 (+ 1416us) server_negotiation.cc:299] Negotiation successful
0929 20:28:52.080293 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":327,"threads_started":1}
I20260929 20:28:52.082810 30871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:52.050849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:34953 (local address 127.28.244.129:48305)
0929 20:28:52.051913 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:52.051976 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:52.054934 (+ 2958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:52.056100 (+ 1166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:52.056109 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:52.057230 (+ 1121us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:28:52.058373 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:52.058411 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:52.060179 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:52.060192 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:52.063352 (+ 3160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:52.063379 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:52.064525 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:52.064536 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:52.064948 (+ 412us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:52.065935 (+ 987us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:28:52.065986 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:28:52.070333 (+ 4347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:28:52.075632 (+ 5299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:52.075645 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:52.075669 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:52.076594 (+ 925us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:52.077243 (+ 649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:52.077251 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:52.077258 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:52.077671 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:52.078534 (+ 863us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:28:52.078548 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:28:52.079296 (+ 748us) client_negotiation.cc:769] Sending connection context
0929 20:28:52.079746 (+ 450us) client_negotiation.cc:240] Negotiation successful
0929 20:28:52.080628 (+ 882us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"thread_start_us":436,"threads_started":1}
I20260929 20:28:52.097612 30846 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.129:34459
I20260929 20:28:52.097723 30981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.129:34459 every 8 connection(s)
I20260929 20:28:52.099452 30846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
I20260929 20:28:52.102051 30982 heartbeater.cc:347] Connected to a master server at 127.28.244.190:34953
I20260929 20:28:52.102324 30982 heartbeater.cc:464] Registering TS with master...
I20260929 20:28:52.103050 30982 heartbeater.cc:511] Master 127.28.244.190:34953 requested a full tablet report, sending...
I20260929 20:28:52.105623 30775 ts_manager.cc:194] Registered new tserver with Master: ee58c80790c7457cbae8da3c197e1a8b (127.28.244.129:34459)
I20260929 20:28:52.106611 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 30846
I20260929 20:28:52.106833 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance
I20260929 20:28:52.107446 30965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:52.091209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:34953 (local address 127.28.244.129:43681)
0929 20:28:52.091794 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:52.091861 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:52.092067 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:52.092786 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:52.092794 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:52.093689 (+ 895us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:28:52.094255 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:52.094273 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:52.095921 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:52.095925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:52.097802 (+ 1877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:52.097825 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:52.098821 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:52.098826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:52.098957 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:52.099724 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:28:52.099743 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:28:52.100962 (+ 1219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:28:52.104652 (+ 3690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:52.104657 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:52.104661 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:52.105219 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:52.105660 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:52.105666 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:52.105669 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:52.105827 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:52.106415 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:28:52.106421 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:28:52.106617 (+ 196us) client_negotiation.cc:769] Sending connection context
0929 20:28:52.106773 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 20:28:52.107124 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":334,"threads_started":1}
I20260929 20:28:52.107623 29650 external_mini_cluster.cc:1428] Setting key 1e0ec8d6095e4630d06b202c232b6995
I20260929 20:28:52.107965 30775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.129@KRBTEST.COM'} at 127.28.244.129:48305
I20260929 20:28:52.108532 30870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:52.091038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:43681 (local address 127.28.244.190:34953)
0929 20:28:52.091263 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:52.091279 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:52.091989 (+ 710us) server_negotiation.cc:407] Connection header received
0929 20:28:52.092271 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:52.092279 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:52.092398 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:52.092520 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:52.094519 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:52.095729 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:52.097974 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:52.098625 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:52.098780 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:52.101265 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:52.101272 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:52.101277 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:52.101320 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:52.104499 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:52.105373 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:52.105377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:52.105379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:52.105535 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:52.105947 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:52.105952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:52.105956 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:52.106287 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:52.106461 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:52.107929 (+ 1468us) server_negotiation.cc:299] Negotiation successful
0929 20:28:52.108168 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
WARNING: no policy specified for kudu/127.28.244.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab.
Entry for principal kudu/127.28.244.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab.
I20260929 20:28:52.152360 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.130:0
--local_ip_for_outbound_sockets=127.28.244.130
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:34953
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:33251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:28:52.506711 30986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:28:52.507045 30986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:28:52.507225 30986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:28:52.515308 30986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:28:52.515437 30986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:28:52.515626 30986 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:28:52.515769 30986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.130
I20260929 20:28:52.527105 30986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:33251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.244.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.244.190:34953
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.130
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:28:52.528661 30986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:28:52.530171 30986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:28:52.545512 30991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:52.545471 30994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:52.545471 30992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:28:52.546432 30986 server_base.cc:1034] running on GCE node
I20260929 20:28:52.547078 30986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:28:52.548285 30986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:28:52.549540 30986 hybrid_clock.cc:648] HybridClock initialized: now 1790713732549453 us; error 68 us; skew 500 ppm
I20260929 20:28:52.549903 30986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:28:52 dist-test-slave-p29l krb5kdc[30730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713732, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:28:52.553526 30986 init.cc:377] Logged in from keytab as kudu/127.28.244.130@KRBTEST.COM (short username kudu)
I20260929 20:28:52.555219 30986 webserver.cc:460] Webserver started at http://127.28.244.130:35565/ using document root <none> and password file <none>
I20260929 20:28:52.556124 30986 fs_manager.cc:359] Metadata directory not provided
I20260929 20:28:52.556264 30986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:28:52.556610 30986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:28:52.559911 30986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-1/data/instance:
uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3"
format_stamp: "Formatted at 2026-09-29 20:28:52 on dist-test-slave-p29l"
server_key: "39e851eb9a1b7e04e14ded7dfaaff662"
server_key_iv: "f4130e022b1d5b99af651cc8bb9fb28a"
server_key_version: "encryptionkey@0"
I20260929 20:28:52.560818 30986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance:
uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3"
format_stamp: "Formatted at 2026-09-29 20:28:52 on dist-test-slave-p29l"
server_key: "39e851eb9a1b7e04e14ded7dfaaff662"
server_key_iv: "f4130e022b1d5b99af651cc8bb9fb28a"
server_key_version: "encryptionkey@0"
I20260929 20:28:52.566905 30986 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 20:28:52.571063 31001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:52.572793 30986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:28:52.573025 30986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-1/wal
uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3"
format_stamp: "Formatted at 2026-09-29 20:28:52 on dist-test-slave-p29l"
server_key: "39e851eb9a1b7e04e14ded7dfaaff662"
server_key_iv: "f4130e022b1d5b99af651cc8bb9fb28a"
server_key_version: "encryptionkey@0"
I20260929 20:28:52.573287 30986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:28:52.619736 30986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:28:52.624979 30986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:28:52.625313 30986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:28:52.628420 30986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:28:52.628543 30986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:52.628703 30986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:28:52.628783 30986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:28:52 dist-test-slave-p29l krb5kdc[30730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713732, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:28:52.656378 31010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:52.629310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:54973 (local address 127.28.244.190:34953)
0929 20:28:52.629820 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:52.629827 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:52.631382 (+ 1555us) server_negotiation.cc:407] Connection header received
0929 20:28:52.634006 (+ 2624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:52.634012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:52.634121 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:52.634253 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:52.637483 (+ 3230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:52.638329 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:52.642082 (+ 3753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:52.642640 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:52.642812 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:52.648742 (+ 5930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:52.648748 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:52.648751 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:52.648792 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:52.651562 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:52.652763 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:52.652773 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:52.652776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:52.652977 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:52.653764 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:52.653772 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:52.653776 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:52.654124 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:52.654314 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:52.655597 (+ 1283us) server_negotiation.cc:299] Negotiation successful
0929 20:28:52.655904 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":253,"threads_started":1}
I20260929 20:28:52.658447 31011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:52.630071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:34953 (local address 127.28.244.130:54973)
0929 20:28:52.631123 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:52.631200 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:52.633564 (+ 2364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:52.634629 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:52.634652 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:52.635792 (+ 1140us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:28:52.637172 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:52.637218 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:52.638532 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:52.638544 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:52.641773 (+ 3229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:52.641808 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:52.642983 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:52.642993 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:52.643604 (+ 611us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:52.644743 (+ 1139us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:28:52.644813 (+ 70us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:28:52.648443 (+ 3630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:28:52.651755 (+ 3312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:52.651766 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:52.651788 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:52.652594 (+ 806us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:52.653130 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:52.653138 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:52.653149 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:52.653600 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:52.654290 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:28:52.654300 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:28:52.655013 (+ 713us) client_negotiation.cc:769] Sending connection context
0929 20:28:52.655467 (+ 454us) client_negotiation.cc:240] Negotiation successful
0929 20:28:52.656363 (+ 896us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":413,"threads_started":1}
I20260929 20:28:52.683422 31010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:52.664667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:60819 (local address 127.28.244.190:34953)
0929 20:28:52.664909 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:52.664916 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:52.665740 (+ 824us) server_negotiation.cc:407] Connection header received
0929 20:28:52.666425 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:52.666434 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:52.666537 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:52.666627 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:52.668795 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:52.670178 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:52.672770 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:52.673380 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:52.673704 (+ 324us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:52.676948 (+ 3244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:52.676958 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:52.676964 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:52.677020 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:52.680230 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:52.681295 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:52.681299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:52.681302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:52.681424 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:52.681823 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:52.681830 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:52.681841 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:52.682215 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:52.682393 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:52.682819 (+ 426us) server_negotiation.cc:299] Negotiation successful
0929 20:28:52.683063 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:28:52.683526 31078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:52.664774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:34953 (local address 127.28.244.130:60819)
0929 20:28:52.665596 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:52.665622 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:52.666217 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:52.666919 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:52.666924 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:52.667764 (+ 840us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:28:52.668549 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:52.668572 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:52.670425 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:52.670434 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:52.672568 (+ 2134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:52.672584 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:52.674168 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:52.674174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:52.674442 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:52.675243 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:28:52.675274 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:28:52.676637 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:28:52.680472 (+ 3835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:52.680480 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:52.680486 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:52.681108 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:52.681549 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:52.681554 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:52.681558 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:52.681700 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:52.682354 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:28:52.682359 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:28:52.682558 (+ 199us) client_negotiation.cc:769] Sending connection context
0929 20:28:52.682710 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 20:28:52.683071 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":426,"threads_started":1}
I20260929 20:28:52.686372 30986 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.130:34117
I20260929 20:28:52.686394 31121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.130:34117 every 8 connection(s)
I20260929 20:28:52.687695 30986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
I20260929 20:28:52.690038 31122 heartbeater.cc:347] Connected to a master server at 127.28.244.190:34953
I20260929 20:28:52.690282 31122 heartbeater.cc:464] Registering TS with master...
I20260929 20:28:52.690975 31122 heartbeater.cc:511] Master 127.28.244.190:34953 requested a full tablet report, sending...
I20260929 20:28:52.692567 30774 ts_manager.cc:194] Registered new tserver with Master: 5c4c758e48154c1aa66eaa01f5cf5eb3 (127.28.244.130:34117)
I20260929 20:28:52.693507 30774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.130@KRBTEST.COM'} at 127.28.244.130:54973
I20260929 20:28:52.694479 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 30986
I20260929 20:28:52.694689 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance
I20260929 20:28:52.695125 29650 external_mini_cluster.cc:1428] Setting key 13c27bc1b031542ecb67c757d085dc48
WARNING: no policy specified for kudu/127.28.244.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab.
Entry for principal kudu/127.28.244.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab.
I20260929 20:28:52.735569 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.131:0
--local_ip_for_outbound_sockets=127.28.244.131
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:34953
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:33251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:28:53.088370 31126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:28:53.088692 31126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:28:53.088889 31126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:28:53.096313 31126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:28:53.096436 31126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:28:53.096596 31126 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:28:53.096688 31126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.131
I20260929 20:28:53.106833 31126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:33251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.244.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.244.190:34953
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.131
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:28:53.108219 31126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:28:53.109592 31126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 20:28:53.113027 30982 heartbeater.cc:503] Master 127.28.244.190:34953 was elected leader, sending a full tablet report...
W20260929 20:28:53.123768 31131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:53.123679 31132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:53.123720 31134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:28:53.124962 31126 server_base.cc:1034] running on GCE node
I20260929 20:28:53.125656 31126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:28:53.126689 31126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:28:53.127925 31126 hybrid_clock.cc:648] HybridClock initialized: now 1790713733127866 us; error 36 us; skew 500 ppm
I20260929 20:28:53.128253 31126 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:28:53 dist-test-slave-p29l krb5kdc[30730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713733, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:28:53.131704 31126 init.cc:377] Logged in from keytab as kudu/127.28.244.131@KRBTEST.COM (short username kudu)
I20260929 20:28:53.133281 31126 webserver.cc:460] Webserver started at http://127.28.244.131:38199/ using document root <none> and password file <none>
I20260929 20:28:53.134048 31126 fs_manager.cc:359] Metadata directory not provided
I20260929 20:28:53.134188 31126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:28:53.134539 31126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:28:53.137666 31126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-2/data/instance:
uuid: "29c658233fab44e0b93c69bfef14da71"
format_stamp: "Formatted at 2026-09-29 20:28:53 on dist-test-slave-p29l"
server_key: "998ff42395ad1d8f1a1cf61ac5b013ac"
server_key_iv: "469b38e7d724ba1edba88ae79c53819e"
server_key_version: "encryptionkey@0"
I20260929 20:28:53.138546 31126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance:
uuid: "29c658233fab44e0b93c69bfef14da71"
format_stamp: "Formatted at 2026-09-29 20:28:53 on dist-test-slave-p29l"
server_key: "998ff42395ad1d8f1a1cf61ac5b013ac"
server_key_iv: "469b38e7d724ba1edba88ae79c53819e"
server_key_version: "encryptionkey@0"
I20260929 20:28:53.144253 31126 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 20:28:53.147976 31141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:53.149458 31126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:28:53.149652 31126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-2/wal
uuid: "29c658233fab44e0b93c69bfef14da71"
format_stamp: "Formatted at 2026-09-29 20:28:53 on dist-test-slave-p29l"
server_key: "998ff42395ad1d8f1a1cf61ac5b013ac"
server_key_iv: "469b38e7d724ba1edba88ae79c53819e"
server_key_version: "encryptionkey@0"
I20260929 20:28:53.149878 31126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:28:53.164506 31126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:28:53.169076 31126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:28:53.169375 31126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:28:53.172302 31126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:28:53.172420 31126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:53.172576 31126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:28:53.172652 31126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:28:53 dist-test-slave-p29l krb5kdc[30730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713733, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:28:53.200143 31010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:53.173122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:43607 (local address 127.28.244.190:34953)
0929 20:28:53.173315 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:53.173321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:53.175093 (+ 1772us) server_negotiation.cc:407] Connection header received
0929 20:28:53.178434 (+ 3341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:53.178444 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:53.178559 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:53.178699 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:53.181394 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.182539 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.185667 (+ 3128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.186626 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.186803 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:53.191896 (+ 5093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:53.191902 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:53.191906 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:53.191972 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:53.195740 (+ 3768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:53.197112 (+ 1372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:53.197116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:53.197119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:53.197259 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:53.197888 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:53.197891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:53.197893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:53.198202 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:53.198318 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:53.199514 (+ 1196us) server_negotiation.cc:299] Negotiation successful
0929 20:28:53.199694 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 20:28:53.202296 31150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:53.173875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:34953 (local address 127.28.244.131:43607)
0929 20:28:53.174854 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:53.174917 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:53.177960 (+ 3043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:53.178986 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:53.179010 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:53.180163 (+ 1153us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:28:53.181179 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:53.181208 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.182735 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.182765 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:53.185435 (+ 2670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:53.185459 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.186935 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.186941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:53.187151 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:53.187973 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:28:53.188008 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:28:53.191530 (+ 3522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:28:53.196013 (+ 4483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:53.196029 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:53.196059 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:53.196928 (+ 869us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:53.197381 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:53.197388 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:53.197394 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:53.197756 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:53.198351 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:28:53.198364 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:28:53.199018 (+ 654us) client_negotiation.cc:769] Sending connection context
0929 20:28:53.199414 (+ 396us) client_negotiation.cc:240] Negotiation successful
0929 20:28:53.200212 (+ 798us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":396,"threads_started":1}
I20260929 20:28:53.224061 31126 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.131:44105
I20260929 20:28:53.224104 31260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.131:44105 every 8 connection(s)
I20260929 20:28:53.225515 31126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
I20260929 20:28:53.226390 31010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:53.211076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:59731 (local address 127.28.244.190:34953)
0929 20:28:53.211259 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:53.211264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:53.212099 (+ 835us) server_negotiation.cc:407] Connection header received
0929 20:28:53.212403 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:53.212408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:53.212489 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:53.212600 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:53.213604 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.214364 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.215799 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.216375 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.216516 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:53.219476 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:53.219481 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:53.219484 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:53.219524 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:53.223027 (+ 3503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:53.224018 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:53.224029 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:53.224033 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:53.224217 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:53.224682 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:53.224687 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:53.224691 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:53.225090 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:53.225253 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:53.225756 (+ 503us) server_negotiation.cc:299] Negotiation successful
0929 20:28:53.226015 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:28:53.226529 31236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:53.211117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:34953 (local address 127.28.244.131:59731)
0929 20:28:53.211961 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:53.211987 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:53.212213 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:53.212627 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:53.212632 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:53.213124 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:28:53.213432 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:53.213443 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.214516 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.214528 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:53.215642 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:53.215656 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.216767 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.216804 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:53.217074 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:53.217839 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:28:53.217871 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:28:53.219163 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:28:53.223320 (+ 4157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:53.223326 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:53.223330 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:53.223861 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:53.224390 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:53.224395 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:53.224399 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:53.224548 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:53.225216 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:28:53.225221 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:28:53.225464 (+ 243us) client_negotiation.cc:769] Sending connection context
0929 20:28:53.225640 (+ 176us) client_negotiation.cc:240] Negotiation successful
0929 20:28:53.226018 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":338,"threads_started":1}
I20260929 20:28:53.228015 31261 heartbeater.cc:347] Connected to a master server at 127.28.244.190:34953
I20260929 20:28:53.230386 31261 heartbeater.cc:464] Registering TS with master...
I20260929 20:28:53.231310 31261 heartbeater.cc:511] Master 127.28.244.190:34953 requested a full tablet report, sending...
I20260929 20:28:53.232970 30774 ts_manager.cc:194] Registered new tserver with Master: 29c658233fab44e0b93c69bfef14da71 (127.28.244.131:44105)
I20260929 20:28:53.233839 30774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.131@KRBTEST.COM'} at 127.28.244.131:43607
I20260929 20:28:53.235355 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 31126
I20260929 20:28:53.235580 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance
I20260929 20:28:53.236076 29650 external_mini_cluster.cc:1428] Setting key b3a5de09bf8737a53036dc30ef9a3986
I20260929 20:28:53.239019 29650 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 20:28:53 dist-test-slave-p29l krb5kdc[30730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713733, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 20:28:53 dist-test-slave-p29l krb5kdc[30730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713733, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:28:53.287520 31010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:53.272608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55856 (local address 127.28.244.190:34953)
0929 20:28:53.272861 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:53.272870 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:53.273105 (+ 235us) server_negotiation.cc:407] Connection header received
0929 20:28:53.273330 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:53.273340 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:53.273405 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:53.273519 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:53.274999 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.275748 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.277160 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.277652 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.277793 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:53.281941 (+ 4148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:53.281947 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:53.281951 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:53.282008 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:53.284546 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:53.285294 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:53.285302 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:53.285308 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:53.285457 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:53.285936 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:53.285942 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:53.285945 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:53.286283 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:53.286605 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:53.286933 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 20:28:53.287175 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 20:28:53.292171 30774 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:28:53.299335 30774 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:28:53.334455 31275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:53.312309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44320 (local address 127.28.244.130:34117)
0929 20:28:53.324599 (+ 12290us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:53.324609 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:53.324643 (+ 34us) server_negotiation.cc:407] Connection header received
0929 20:28:53.324764 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:53.324774 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:53.325086 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:53.325782 (+ 696us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:28:53.326736 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.328347 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.331665 (+ 3318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.333186 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.333368 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:53.333564 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:53.333964 (+ 400us) server_negotiation.cc:299] Negotiation successful
0929 20:28:53.334144 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12100,"thread_start_us":12959,"threads_started":1}
I20260929 20:28:53.334607 31274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:53.312115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:34117 (local address 127.0.0.1:44320)
0929 20:28:53.313077 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:53.313120 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:53.313324 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:53.325607 (+ 12283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:53.325617 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:53.325676 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:28:53.326545 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:53.326556 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.328538 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.328543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:53.331480 (+ 2937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:53.331504 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.333403 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.333411 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:53.333663 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:53.333784 (+ 121us) client_negotiation.cc:769] Sending connection context
0929 20:28:53.333915 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 20:28:53.334142 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"mutex_wait_us":909,"spinlock_wait_cycles":13056,"thread_start_us":535,"threads_started":1}
I20260929 20:28:53.335027 31277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:53.313391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43578 (local address 127.28.244.129:34459)
0929 20:28:53.324599 (+ 11208us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:53.324609 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:53.324643 (+ 34us) server_negotiation.cc:407] Connection header received
0929 20:28:53.324764 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:53.324774 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:53.325086 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:53.326776 (+ 1690us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:28:53.327012 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.328275 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.331700 (+ 3425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.333247 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.334295 (+ 1048us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:53.334479 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:53.334554 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 20:28:53.334747 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11020,"thread_start_us":10906,"threads_started":1}
I20260929 20:28:53.335290 31279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:53.314150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49634 (local address 127.28.244.131:44105)
0929 20:28:53.326230 (+ 12080us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:53.326237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:53.326265 (+ 28us) server_negotiation.cc:407] Connection header received
0929 20:28:53.326347 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:53.326352 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:53.326571 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:53.327520 (+ 949us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:28:53.327854 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.329152 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.331505 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.333186 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.333387 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:53.333564 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:53.334797 (+ 1233us) server_negotiation.cc:299] Negotiation successful
0929 20:28:53.334962 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11912,"thread_start_us":11733,"threads_started":1}
I20260929 20:28:53.334669 31278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:53.313861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:44105 (local address 127.0.0.1:49634)
0929 20:28:53.325184 (+ 11323us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:53.325213 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:53.325392 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:53.326826 (+ 1434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:53.326831 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:53.326861 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:28:53.327298 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:53.327312 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.329309 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.329315 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:53.331318 (+ 2003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:53.331332 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.333827 (+ 2495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.333834 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:53.334008 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:53.334041 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 20:28:53.334120 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 20:28:53.334236 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11129,"mutex_wait_us":162,"thread_start_us":11377,"threads_started":1}
I20260929 20:28:53.335068 31276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:53.312777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:34459 (local address 127.0.0.1:43578)
0929 20:28:53.314029 (+ 1252us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:53.314050 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:53.314160 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:53.325607 (+ 11447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:53.325617 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:53.325675 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:28:53.326107 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:53.326119 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.328418 (+ 2299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.328425 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:53.331513 (+ 3088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:53.331522 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.333513 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.333518 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:53.333687 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:53.333748 (+ 61us) client_negotiation.cc:769] Sending connection context
0929 20:28:53.333879 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 20:28:53.334141 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1098,"mutex_wait_us":224,"spinlock_wait_cycles":7680,"thread_start_us":1485,"threads_started":1}
I20260929 20:28:53.339076 30912 tablet_service.cc:1467] Processing CreateTablet for tablet 40bf658695f741d9a51117dafe6fe5c4 (DEFAULT_TABLE table=test-table [id=027d6bf054e94c6bb4864112f19afa85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:28:53.339201 31191 tablet_service.cc:1467] Processing CreateTablet for tablet 40bf658695f741d9a51117dafe6fe5c4 (DEFAULT_TABLE table=test-table [id=027d6bf054e94c6bb4864112f19afa85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:28:53.339200 31052 tablet_service.cc:1467] Processing CreateTablet for tablet 40bf658695f741d9a51117dafe6fe5c4 (DEFAULT_TABLE table=test-table [id=027d6bf054e94c6bb4864112f19afa85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:28:53.340669 30912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40bf658695f741d9a51117dafe6fe5c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:28:53.340684 31191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40bf658695f741d9a51117dafe6fe5c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:28:53.340687 31052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40bf658695f741d9a51117dafe6fe5c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:28:53.357779 31280 tablet_bootstrap.cc:492] T 40bf658695f741d9a51117dafe6fe5c4 P ee58c80790c7457cbae8da3c197e1a8b: Bootstrap starting.
I20260929 20:28:53.360023 31282 tablet_bootstrap.cc:492] T 40bf658695f741d9a51117dafe6fe5c4 P 29c658233fab44e0b93c69bfef14da71: Bootstrap starting.
I20260929 20:28:53.362120 31280 tablet_bootstrap.cc:654] T 40bf658695f741d9a51117dafe6fe5c4 P ee58c80790c7457cbae8da3c197e1a8b: Neither blocks nor log segments found. Creating new log.
I20260929 20:28:53.363539 31280 log.cc:824] T 40bf658695f741d9a51117dafe6fe5c4 P ee58c80790c7457cbae8da3c197e1a8b: Log is configured to *not* fsync() on all Append() calls
I20260929 20:28:53.363749 31282 tablet_bootstrap.cc:654] T 40bf658695f741d9a51117dafe6fe5c4 P 29c658233fab44e0b93c69bfef14da71: Neither blocks nor log segments found. Creating new log.
I20260929 20:28:53.365157 31282 log.cc:824] T 40bf658695f741d9a51117dafe6fe5c4 P 29c658233fab44e0b93c69bfef14da71: Log is configured to *not* fsync() on all Append() calls
I20260929 20:28:53.368132 31280 tablet_bootstrap.cc:492] T 40bf658695f741d9a51117dafe6fe5c4 P ee58c80790c7457cbae8da3c197e1a8b: No bootstrap required, opened a new log
I20260929 20:28:53.368513 31280 ts_tablet_manager.cc:1397] T 40bf658695f741d9a51117dafe6fe5c4 P ee58c80790c7457cbae8da3c197e1a8b: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260929 20:28:53.372697 31281 tablet_bootstrap.cc:492] T 40bf658695f741d9a51117dafe6fe5c4 P 5c4c758e48154c1aa66eaa01f5cf5eb3: Bootstrap starting.
I20260929 20:28:53.374693 31282 tablet_bootstrap.cc:492] T 40bf658695f741d9a51117dafe6fe5c4 P 29c658233fab44e0b93c69bfef14da71: No bootstrap required, opened a new log
I20260929 20:28:53.375025 31282 ts_tablet_manager.cc:1397] T 40bf658695f741d9a51117dafe6fe5c4 P 29c658233fab44e0b93c69bfef14da71: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260929 20:28:53.376724 31281 tablet_bootstrap.cc:654] T 40bf658695f741d9a51117dafe6fe5c4 P 5c4c758e48154c1aa66eaa01f5cf5eb3: Neither blocks nor log segments found. Creating new log.
I20260929 20:28:53.378153 31281 log.cc:824] T 40bf658695f741d9a51117dafe6fe5c4 P 5c4c758e48154c1aa66eaa01f5cf5eb3: Log is configured to *not* fsync() on all Append() calls
I20260929 20:28:53.381206 31280 raft_consensus.cc:348] T 40bf658695f741d9a51117dafe6fe5c4 P ee58c80790c7457cbae8da3c197e1a8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 } } peers { permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 } } peers { permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 } }
I20260929 20:28:53.381685 31280 raft_consensus.cc:374] T 40bf658695f741d9a51117dafe6fe5c4 P ee58c80790c7457cbae8da3c197e1a8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:28:53.381821 31280 raft_consensus.cc:729] T 40bf658695f741d9a51117dafe6fe5c4 P ee58c80790c7457cbae8da3c197e1a8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee58c80790c7457cbae8da3c197e1a8b, State: Initialized, Role: FOLLOWER
I20260929 20:28:53.381968 31281 tablet_bootstrap.cc:492] T 40bf658695f741d9a51117dafe6fe5c4 P 5c4c758e48154c1aa66eaa01f5cf5eb3: No bootstrap required, opened a new log
I20260929 20:28:53.382256 31281 ts_tablet_manager.cc:1397] T 40bf658695f741d9a51117dafe6fe5c4 P 5c4c758e48154c1aa66eaa01f5cf5eb3: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260929 20:28:53.382532 31280 consensus_queue.cc:260] T 40bf658695f741d9a51117dafe6fe5c4 P ee58c80790c7457cbae8da3c197e1a8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 } } peers { permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 } } peers { permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 } }
I20260929 20:28:53.384490 31280 ts_tablet_manager.cc:1428] T 40bf658695f741d9a51117dafe6fe5c4 P ee58c80790c7457cbae8da3c197e1a8b: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260929 20:28:53.387915 31282 raft_consensus.cc:348] T 40bf658695f741d9a51117dafe6fe5c4 P 29c658233fab44e0b93c69bfef14da71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 } } peers { permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 } } peers { permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 } }
I20260929 20:28:53.388501 31282 raft_consensus.cc:374] T 40bf658695f741d9a51117dafe6fe5c4 P 29c658233fab44e0b93c69bfef14da71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:28:53.388664 31282 raft_consensus.cc:729] T 40bf658695f741d9a51117dafe6fe5c4 P 29c658233fab44e0b93c69bfef14da71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29c658233fab44e0b93c69bfef14da71, State: Initialized, Role: FOLLOWER
I20260929 20:28:53.388633 31286 raft_consensus.cc:482] T 40bf658695f741d9a51117dafe6fe5c4 P ee58c80790c7457cbae8da3c197e1a8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:28:53.388988 31286 raft_consensus.cc:504] T 40bf658695f741d9a51117dafe6fe5c4 P ee58c80790c7457cbae8da3c197e1a8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 } } peers { permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 } } peers { permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 } }
I20260929 20:28:53.389550 31282 consensus_queue.cc:260] T 40bf658695f741d9a51117dafe6fe5c4 P 29c658233fab44e0b93c69bfef14da71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 } } peers { permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 } } peers { permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 } }
I20260929 20:28:53.393019 31281 raft_consensus.cc:348] T 40bf658695f741d9a51117dafe6fe5c4 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 } } peers { permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 } } peers { permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 } }
I20260929 20:28:53.393527 31281 raft_consensus.cc:374] T 40bf658695f741d9a51117dafe6fe5c4 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:28:53.393656 31281 raft_consensus.cc:729] T 40bf658695f741d9a51117dafe6fe5c4 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c4c758e48154c1aa66eaa01f5cf5eb3, State: Initialized, Role: FOLLOWER
I20260929 20:28:53.394325 31281 consensus_queue.cc:260] T 40bf658695f741d9a51117dafe6fe5c4 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 } } peers { permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 } } peers { permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 } }
I20260929 20:28:53.395498 31122 heartbeater.cc:503] Master 127.28.244.190:34953 was elected leader, sending a full tablet report...
I20260929 20:28:53.396312 31281 ts_tablet_manager.cc:1428] T 40bf658695f741d9a51117dafe6fe5c4 P 5c4c758e48154c1aa66eaa01f5cf5eb3: Time spent starting tablet: real 0.014s user 0.007s sys 0.008s
I20260929 20:28:53.396921 31286 leader_election.cc:288] T 40bf658695f741d9a51117dafe6fe5c4 P ee58c80790c7457cbae8da3c197e1a8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c4c758e48154c1aa66eaa01f5cf5eb3 (127.28.244.130:34117), 29c658233fab44e0b93c69bfef14da71 (127.28.244.131:44105)
I20260929 20:28:53.398679 31261 heartbeater.cc:503] Master 127.28.244.190:34953 was elected leader, sending a full tablet report...
I20260929 20:28:53.399775 31282 ts_tablet_manager.cc:1428] T 40bf658695f741d9a51117dafe6fe5c4 P 29c658233fab44e0b93c69bfef14da71: Time spent starting tablet: real 0.025s user 0.014s sys 0.011s
I20260929 20:28:53.408104 31275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:53.394216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:58621 (local address 127.28.244.130:34117)
0929 20:28:53.394431 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:53.394437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:53.398357 (+ 3920us) server_negotiation.cc:407] Connection header received
0929 20:28:53.398430 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:53.398434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:53.398521 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:53.398614 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:28:53.401947 (+ 3333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.403006 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.405657 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.406863 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.407080 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:53.407174 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:53.407621 (+ 447us) server_negotiation.cc:299] Negotiation successful
0929 20:28:53.407770 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 20:28:53.408285 31288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:53.393543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:34117 (local address 127.28.244.129:58621)
0929 20:28:53.397654 (+ 4111us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:53.397675 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:53.397833 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:53.401238 (+ 3405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:53.401243 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:53.401277 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:28:53.401740 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:53.401781 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.403208 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.403213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:53.405488 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:53.405503 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.407137 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.407146 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:53.407373 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:53.407458 (+ 85us) client_negotiation.cc:769] Sending connection context
0929 20:28:53.407578 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 20:28:53.407744 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3961,"mutex_wait_us":32,"thread_start_us":3768,"threads_started":1}
I20260929 20:28:53.409700 31076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40bf658695f741d9a51117dafe6fe5c4" candidate_uuid: "ee58c80790c7457cbae8da3c197e1a8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" is_pre_election: true
I20260929 20:28:53.410358 31076 raft_consensus.cc:2454] T 40bf658695f741d9a51117dafe6fe5c4 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee58c80790c7457cbae8da3c197e1a8b in term 0.
I20260929 20:28:53.411319 31290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:53.398501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:44105 (local address 127.28.244.129:37557)
0929 20:28:53.404444 (+ 5943us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:53.404464 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:53.404609 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:53.405021 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:53.405307 (+ 286us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:53.405347 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:28:53.405773 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:53.405782 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.406998 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.407006 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:53.408760 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:53.408776 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.410552 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.410559 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:53.410723 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:53.410750 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 20:28:53.410872 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 20:28:53.411013 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5773,"mutex_wait_us":16,"thread_start_us":5566,"threads_started":1}
I20260929 20:28:53.411366 31279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:53.398118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:37557 (local address 127.28.244.131:44105)
0929 20:28:53.398336 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:53.398342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:53.404555 (+ 6213us) server_negotiation.cc:407] Connection header received
0929 20:28:53.404725 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:53.404731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:53.404871 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:53.404972 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:28:53.406023 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.406855 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.408939 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.410315 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.410515 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:53.410597 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:53.410937 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 20:28:53.411086 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:28:53.411213 30862 leader_election.cc:302] T 40bf658695f741d9a51117dafe6fe5c4 P ee58c80790c7457cbae8da3c197e1a8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c4c758e48154c1aa66eaa01f5cf5eb3, ee58c80790c7457cbae8da3c197e1a8b; no voters:
I20260929 20:28:53.411718 31286 raft_consensus.cc:2781] T 40bf658695f741d9a51117dafe6fe5c4 P ee58c80790c7457cbae8da3c197e1a8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:28:53.411907 31286 raft_consensus.cc:482] T 40bf658695f741d9a51117dafe6fe5c4 P ee58c80790c7457cbae8da3c197e1a8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:28:53.412021 31286 raft_consensus.cc:3037] T 40bf658695f741d9a51117dafe6fe5c4 P ee58c80790c7457cbae8da3c197e1a8b [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:28:53.412611 31211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40bf658695f741d9a51117dafe6fe5c4" candidate_uuid: "ee58c80790c7457cbae8da3c197e1a8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29c658233fab44e0b93c69bfef14da71" is_pre_election: true
I20260929 20:28:53.413170 31211 raft_consensus.cc:2454] T 40bf658695f741d9a51117dafe6fe5c4 P 29c658233fab44e0b93c69bfef14da71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee58c80790c7457cbae8da3c197e1a8b in term 0.
I20260929 20:28:53.414793 31286 raft_consensus.cc:504] T 40bf658695f741d9a51117dafe6fe5c4 P ee58c80790c7457cbae8da3c197e1a8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 } } peers { permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 } } peers { permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 } }
I20260929 20:28:53.415323 31286 leader_election.cc:288] T 40bf658695f741d9a51117dafe6fe5c4 P ee58c80790c7457cbae8da3c197e1a8b [CANDIDATE]: Term 1 election: Requested vote from peers 5c4c758e48154c1aa66eaa01f5cf5eb3 (127.28.244.130:34117), 29c658233fab44e0b93c69bfef14da71 (127.28.244.131:44105)
I20260929 20:28:53.415989 31211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40bf658695f741d9a51117dafe6fe5c4" candidate_uuid: "ee58c80790c7457cbae8da3c197e1a8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29c658233fab44e0b93c69bfef14da71"
I20260929 20:28:53.415994 31076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40bf658695f741d9a51117dafe6fe5c4" candidate_uuid: "ee58c80790c7457cbae8da3c197e1a8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3"
I20260929 20:28:53.416209 31211 raft_consensus.cc:3037] T 40bf658695f741d9a51117dafe6fe5c4 P 29c658233fab44e0b93c69bfef14da71 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:28:53.416247 31076 raft_consensus.cc:3037] T 40bf658695f741d9a51117dafe6fe5c4 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:28:53.418661 31211 raft_consensus.cc:2454] T 40bf658695f741d9a51117dafe6fe5c4 P 29c658233fab44e0b93c69bfef14da71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee58c80790c7457cbae8da3c197e1a8b in term 1.
I20260929 20:28:53.418823 31076 raft_consensus.cc:2454] T 40bf658695f741d9a51117dafe6fe5c4 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee58c80790c7457cbae8da3c197e1a8b in term 1.
I20260929 20:28:53.419102 30862 leader_election.cc:302] T 40bf658695f741d9a51117dafe6fe5c4 P ee58c80790c7457cbae8da3c197e1a8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29c658233fab44e0b93c69bfef14da71, ee58c80790c7457cbae8da3c197e1a8b; no voters:
I20260929 20:28:53.419448 31286 raft_consensus.cc:2781] T 40bf658695f741d9a51117dafe6fe5c4 P ee58c80790c7457cbae8da3c197e1a8b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:28:53.419981 31286 raft_consensus.cc:686] T 40bf658695f741d9a51117dafe6fe5c4 P ee58c80790c7457cbae8da3c197e1a8b [term 1 LEADER]: Becoming Leader. State: Replica: ee58c80790c7457cbae8da3c197e1a8b, State: Running, Role: LEADER
I20260929 20:28:53.420465 31286 consensus_queue.cc:237] T 40bf658695f741d9a51117dafe6fe5c4 P ee58c80790c7457cbae8da3c197e1a8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 } } peers { permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 } } peers { permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 } }
I20260929 20:28:53.425249 30775 catalog_manager.cc:5495] T 40bf658695f741d9a51117dafe6fe5c4 P ee58c80790c7457cbae8da3c197e1a8b reported cstate change: term changed from 0 to 1, leader changed from <none> to ee58c80790c7457cbae8da3c197e1a8b (127.28.244.129). New cstate: current_term: 1 leader_uuid: "ee58c80790c7457cbae8da3c197e1a8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 } health_report { overall_health: HEALTHY } } }
W20260929 20:28:53.439970 31123 tablet.cc:2369] T 40bf658695f741d9a51117dafe6fe5c4 P 5c4c758e48154c1aa66eaa01f5cf5eb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:28:53.470518 31277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:53.464111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43580 (local address 127.28.244.129:34459)
0929 20:28:53.464401 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:53.464409 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:53.464431 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:28:53.464689 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:53.464702 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:53.464786 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:53.464944 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:28:53.465332 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.466079 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.467523 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.468005 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.468140 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:53.468690 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:28:53.469472 (+ 782us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:28:53.469626 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:53.470124 (+ 498us) server_negotiation.cc:299] Negotiation successful
0929 20:28:53.470211 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
W20260929 20:28:53.478653 31262 tablet.cc:2369] T 40bf658695f741d9a51117dafe6fe5c4 P 29c658233fab44e0b93c69bfef14da71: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:28:53.484530 31211 raft_consensus.cc:1261] T 40bf658695f741d9a51117dafe6fe5c4 P 29c658233fab44e0b93c69bfef14da71 [term 1 FOLLOWER]: Refusing update from remote peer ee58c80790c7457cbae8da3c197e1a8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:28:53.484536 31076 raft_consensus.cc:1261] T 40bf658695f741d9a51117dafe6fe5c4 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [term 1 FOLLOWER]: Refusing update from remote peer ee58c80790c7457cbae8da3c197e1a8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:28:53.485644 31291 consensus_queue.cc:1035] T 40bf658695f741d9a51117dafe6fe5c4 P ee58c80790c7457cbae8da3c197e1a8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:28:53.486044 31286 consensus_queue.cc:1035] T 40bf658695f741d9a51117dafe6fe5c4 P ee58c80790c7457cbae8da3c197e1a8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:28:53.497473 31296 mvcc.cc:205] Tried to move back new op lower bound from 7334763452341100544 to 7334763452093972480. Current Snapshot: MvccSnapshot[applied={T|T < 7334763452341100544}]
I20260929 20:28:53.518534 30775 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:28:53.518962 30775 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:28:53.524088 30775 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a039d1b8e4f4eb9849b1bf235f81620: Sending DeleteTablet for 3 replicas of tablet 40bf658695f741d9a51117dafe6fe5c4
I20260929 20:28:53.525525 31191 tablet_service.cc:1514] Processing DeleteTablet for tablet 40bf658695f741d9a51117dafe6fe5c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:28:53 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:49634
I20260929 20:28:53.525645 30912 tablet_service.cc:1514] Processing DeleteTablet for tablet 40bf658695f741d9a51117dafe6fe5c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:28:53 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:43578
I20260929 20:28:53.526321 31052 tablet_service.cc:1514] Processing DeleteTablet for tablet 40bf658695f741d9a51117dafe6fe5c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:28:53 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:44320
I20260929 20:28:53.526569 31305 tablet_replica.cc:330] T 40bf658695f741d9a51117dafe6fe5c4 P ee58c80790c7457cbae8da3c197e1a8b: stopping tablet replica
I20260929 20:28:53.527055 31305 raft_consensus.cc:2229] T 40bf658695f741d9a51117dafe6fe5c4 P ee58c80790c7457cbae8da3c197e1a8b [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:28:53.528604 31304 tablet_replica.cc:330] T 40bf658695f741d9a51117dafe6fe5c4 P 29c658233fab44e0b93c69bfef14da71: stopping tablet replica
I20260929 20:28:53.528892 31305 raft_consensus.cc:2258] T 40bf658695f741d9a51117dafe6fe5c4 P ee58c80790c7457cbae8da3c197e1a8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:28:53.529686 31304 raft_consensus.cc:2229] T 40bf658695f741d9a51117dafe6fe5c4 P 29c658233fab44e0b93c69bfef14da71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:28:53.530181 31304 raft_consensus.cc:2258] T 40bf658695f741d9a51117dafe6fe5c4 P 29c658233fab44e0b93c69bfef14da71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:28:53.532572 31306 tablet_replica.cc:330] T 40bf658695f741d9a51117dafe6fe5c4 P 5c4c758e48154c1aa66eaa01f5cf5eb3: stopping tablet replica
I20260929 20:28:53.532707 31305 ts_tablet_manager.cc:1905] T 40bf658695f741d9a51117dafe6fe5c4 P ee58c80790c7457cbae8da3c197e1a8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:28:53.532572 30774 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 20:28:53.533067 31306 raft_consensus.cc:2229] T 40bf658695f741d9a51117dafe6fe5c4 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260929 20:28:53.533330 30774 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:28:53.533560 31306 raft_consensus.cc:2258] T 40bf658695f741d9a51117dafe6fe5c4 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:28:53.537302 31304 ts_tablet_manager.cc:1905] T 40bf658695f741d9a51117dafe6fe5c4 P 29c658233fab44e0b93c69bfef14da71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:28:53.538908 31306 ts_tablet_manager.cc:1905] T 40bf658695f741d9a51117dafe6fe5c4 P 5c4c758e48154c1aa66eaa01f5cf5eb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:28:53.541954 31052 tablet_service.cc:1467] Processing CreateTablet for tablet 9b2d36624d754de383ad2277b852b176 (DEFAULT_TABLE table=test-table [id=bb592b8eeaec43b591866e8cda7a1c8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:28:53.542153 31191 tablet_service.cc:1467] Processing CreateTablet for tablet 9b2d36624d754de383ad2277b852b176 (DEFAULT_TABLE table=test-table [id=bb592b8eeaec43b591866e8cda7a1c8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:28:53.542173 30912 tablet_service.cc:1467] Processing CreateTablet for tablet 9b2d36624d754de383ad2277b852b176 (DEFAULT_TABLE table=test-table [id=bb592b8eeaec43b591866e8cda7a1c8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:28:53.542558 31052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b2d36624d754de383ad2277b852b176. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:28:53.542558 31191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b2d36624d754de383ad2277b852b176. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:28:53.542642 30912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b2d36624d754de383ad2277b852b176. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:28:53.549242 31281 tablet_bootstrap.cc:492] T 9b2d36624d754de383ad2277b852b176 P 5c4c758e48154c1aa66eaa01f5cf5eb3: Bootstrap starting.
I20260929 20:28:53.549271 31282 tablet_bootstrap.cc:492] T 9b2d36624d754de383ad2277b852b176 P 29c658233fab44e0b93c69bfef14da71: Bootstrap starting.
I20260929 20:28:53.551106 31282 tablet_bootstrap.cc:654] T 9b2d36624d754de383ad2277b852b176 P 29c658233fab44e0b93c69bfef14da71: Neither blocks nor log segments found. Creating new log.
I20260929 20:28:53.551416 31281 tablet_bootstrap.cc:654] T 9b2d36624d754de383ad2277b852b176 P 5c4c758e48154c1aa66eaa01f5cf5eb3: Neither blocks nor log segments found. Creating new log.
I20260929 20:28:53.555557 31280 tablet_bootstrap.cc:492] T 9b2d36624d754de383ad2277b852b176 P ee58c80790c7457cbae8da3c197e1a8b: Bootstrap starting.
I20260929 20:28:53.557344 31280 tablet_bootstrap.cc:654] T 9b2d36624d754de383ad2277b852b176 P ee58c80790c7457cbae8da3c197e1a8b: Neither blocks nor log segments found. Creating new log.
I20260929 20:28:53.567559 31282 tablet_bootstrap.cc:492] T 9b2d36624d754de383ad2277b852b176 P 29c658233fab44e0b93c69bfef14da71: No bootstrap required, opened a new log
I20260929 20:28:53.567756 31282 ts_tablet_manager.cc:1397] T 9b2d36624d754de383ad2277b852b176 P 29c658233fab44e0b93c69bfef14da71: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.009s
I20260929 20:28:53.568754 31282 raft_consensus.cc:348] T 9b2d36624d754de383ad2277b852b176 P 29c658233fab44e0b93c69bfef14da71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 } } peers { permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 } } peers { permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 } }
I20260929 20:28:53.569227 31282 raft_consensus.cc:374] T 9b2d36624d754de383ad2277b852b176 P 29c658233fab44e0b93c69bfef14da71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:28:53.569489 31282 raft_consensus.cc:729] T 9b2d36624d754de383ad2277b852b176 P 29c658233fab44e0b93c69bfef14da71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29c658233fab44e0b93c69bfef14da71, State: Initialized, Role: FOLLOWER
I20260929 20:28:53.569911 31282 consensus_queue.cc:260] T 9b2d36624d754de383ad2277b852b176 P 29c658233fab44e0b93c69bfef14da71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 } } peers { permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 } } peers { permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 } }
I20260929 20:28:53.570509 31304 ts_tablet_manager.cc:1918] T 40bf658695f741d9a51117dafe6fe5c4 P 29c658233fab44e0b93c69bfef14da71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:28:53.570698 31304 log.cc:1196] T 40bf658695f741d9a51117dafe6fe5c4 P 29c658233fab44e0b93c69bfef14da71: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-2/wal/wals/40bf658695f741d9a51117dafe6fe5c4
I20260929 20:28:53.571394 31304 ts_tablet_manager.cc:1939] T 40bf658695f741d9a51117dafe6fe5c4 P 29c658233fab44e0b93c69bfef14da71: Deleting consensus metadata
I20260929 20:28:53.571866 31282 ts_tablet_manager.cc:1428] T 9b2d36624d754de383ad2277b852b176 P 29c658233fab44e0b93c69bfef14da71: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260929 20:28:53.573645 31306 ts_tablet_manager.cc:1918] T 40bf658695f741d9a51117dafe6fe5c4 P 5c4c758e48154c1aa66eaa01f5cf5eb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:28:53.573705 30759 catalog_manager.cc:4841] TS 29c658233fab44e0b93c69bfef14da71 (127.28.244.131:44105): tablet 40bf658695f741d9a51117dafe6fe5c4 (table test-table [id=027d6bf054e94c6bb4864112f19afa85]) successfully deleted
I20260929 20:28:53.573879 31306 log.cc:1196] T 40bf658695f741d9a51117dafe6fe5c4 P 5c4c758e48154c1aa66eaa01f5cf5eb3: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-1/wal/wals/40bf658695f741d9a51117dafe6fe5c4
I20260929 20:28:53.574548 31306 ts_tablet_manager.cc:1939] T 40bf658695f741d9a51117dafe6fe5c4 P 5c4c758e48154c1aa66eaa01f5cf5eb3: Deleting consensus metadata
I20260929 20:28:53.576730 30759 catalog_manager.cc:4841] TS 5c4c758e48154c1aa66eaa01f5cf5eb3 (127.28.244.130:34117): tablet 40bf658695f741d9a51117dafe6fe5c4 (table test-table [id=027d6bf054e94c6bb4864112f19afa85]) successfully deleted
I20260929 20:28:53.582108 31280 tablet_bootstrap.cc:492] T 9b2d36624d754de383ad2277b852b176 P ee58c80790c7457cbae8da3c197e1a8b: No bootstrap required, opened a new log
I20260929 20:28:53.582126 31281 tablet_bootstrap.cc:492] T 9b2d36624d754de383ad2277b852b176 P 5c4c758e48154c1aa66eaa01f5cf5eb3: No bootstrap required, opened a new log
I20260929 20:28:53.582309 31280 ts_tablet_manager.cc:1397] T 9b2d36624d754de383ad2277b852b176 P ee58c80790c7457cbae8da3c197e1a8b: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.004s
I20260929 20:28:53.582435 31281 ts_tablet_manager.cc:1397] T 9b2d36624d754de383ad2277b852b176 P 5c4c758e48154c1aa66eaa01f5cf5eb3: Time spent bootstrapping tablet: real 0.033s user 0.000s sys 0.012s
I20260929 20:28:53.583279 31280 raft_consensus.cc:348] T 9b2d36624d754de383ad2277b852b176 P ee58c80790c7457cbae8da3c197e1a8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 } } peers { permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 } } peers { permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 } }
I20260929 20:28:53.583282 31281 raft_consensus.cc:348] T 9b2d36624d754de383ad2277b852b176 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 } } peers { permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 } } peers { permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 } }
I20260929 20:28:53.583539 31280 raft_consensus.cc:374] T 9b2d36624d754de383ad2277b852b176 P ee58c80790c7457cbae8da3c197e1a8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:28:53.583626 31305 ts_tablet_manager.cc:1918] T 40bf658695f741d9a51117dafe6fe5c4 P ee58c80790c7457cbae8da3c197e1a8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:28:53.583675 31281 raft_consensus.cc:374] T 9b2d36624d754de383ad2277b852b176 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:28:53.583815 31281 raft_consensus.cc:729] T 9b2d36624d754de383ad2277b852b176 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c4c758e48154c1aa66eaa01f5cf5eb3, State: Initialized, Role: FOLLOWER
I20260929 20:28:53.583884 31305 log.cc:1196] T 40bf658695f741d9a51117dafe6fe5c4 P ee58c80790c7457cbae8da3c197e1a8b: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-0/wal/wals/40bf658695f741d9a51117dafe6fe5c4
I20260929 20:28:53.584157 31281 consensus_queue.cc:260] T 9b2d36624d754de383ad2277b852b176 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 } } peers { permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 } } peers { permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 } }
I20260929 20:28:53.584443 31305 ts_tablet_manager.cc:1939] T 40bf658695f741d9a51117dafe6fe5c4 P ee58c80790c7457cbae8da3c197e1a8b: Deleting consensus metadata
I20260929 20:28:53.583650 31280 raft_consensus.cc:729] T 9b2d36624d754de383ad2277b852b176 P ee58c80790c7457cbae8da3c197e1a8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee58c80790c7457cbae8da3c197e1a8b, State: Initialized, Role: FOLLOWER
I20260929 20:28:53.584801 31281 ts_tablet_manager.cc:1428] T 9b2d36624d754de383ad2277b852b176 P 5c4c758e48154c1aa66eaa01f5cf5eb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 20:28:53.585043 31280 consensus_queue.cc:260] T 9b2d36624d754de383ad2277b852b176 P ee58c80790c7457cbae8da3c197e1a8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 } } peers { permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 } } peers { permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 } }
I20260929 20:28:53.586467 30762 catalog_manager.cc:4841] TS ee58c80790c7457cbae8da3c197e1a8b (127.28.244.129:34459): tablet 40bf658695f741d9a51117dafe6fe5c4 (table test-table [id=027d6bf054e94c6bb4864112f19afa85]) successfully deleted
I20260929 20:28:53.586628 31280 ts_tablet_manager.cc:1428] T 9b2d36624d754de383ad2277b852b176 P ee58c80790c7457cbae8da3c197e1a8b: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
W20260929 20:28:53.603157 30983 tablet.cc:2369] T 9b2d36624d754de383ad2277b852b176 P ee58c80790c7457cbae8da3c197e1a8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:28:53.661629 31289 raft_consensus.cc:482] T 9b2d36624d754de383ad2277b852b176 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:28:53.661934 31289 raft_consensus.cc:504] T 9b2d36624d754de383ad2277b852b176 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 } } peers { permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 } } peers { permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 } }
I20260929 20:28:53.664232 31289 leader_election.cc:288] T 9b2d36624d754de383ad2277b852b176 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee58c80790c7457cbae8da3c197e1a8b (127.28.244.129:34459), 29c658233fab44e0b93c69bfef14da71 (127.28.244.131:44105)
I20260929 20:28:53.670455 31287 raft_consensus.cc:482] T 9b2d36624d754de383ad2277b852b176 P 29c658233fab44e0b93c69bfef14da71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:28:53.670738 31287 raft_consensus.cc:504] T 9b2d36624d754de383ad2277b852b176 P 29c658233fab44e0b93c69bfef14da71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 } } peers { permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 } } peers { permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 } }
I20260929 20:28:53.672717 31287 leader_election.cc:288] T 9b2d36624d754de383ad2277b852b176 P 29c658233fab44e0b93c69bfef14da71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee58c80790c7457cbae8da3c197e1a8b (127.28.244.129:34459), 5c4c758e48154c1aa66eaa01f5cf5eb3 (127.28.244.130:34117)
I20260929 20:28:53.674973 31277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:53.664936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:45661 (local address 127.28.244.129:34459)
0929 20:28:53.665174 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:53.665181 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:53.667513 (+ 2332us) server_negotiation.cc:407] Connection header received
0929 20:28:53.667884 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:53.667890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:53.668025 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:53.668128 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:28:53.668765 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.669737 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.672310 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.673643 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.673773 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:53.673863 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:53.674438 (+ 575us) server_negotiation.cc:299] Negotiation successful
0929 20:28:53.674607 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 20:28:53.676292 31279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:53.665057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:57585 (local address 127.28.244.131:44105)
0929 20:28:53.665250 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:53.665260 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:53.668032 (+ 2772us) server_negotiation.cc:407] Connection header received
0929 20:28:53.668215 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:53.668221 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:53.668299 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:53.668361 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:28:53.669204 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.670547 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.673394 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.674852 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.674985 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:53.675066 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:53.675828 (+ 762us) server_negotiation.cc:299] Negotiation successful
0929 20:28:53.675983 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 20:28:53.677199 31310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:53.664559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:44105 (local address 127.28.244.130:57585)
0929 20:28:53.667894 (+ 3335us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:53.667910 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:53.668042 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:53.668532 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:53.668535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:53.668557 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:28:53.668963 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:53.668976 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.670764 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.670770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:53.673188 (+ 2418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:53.673204 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.675275 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.675381 (+ 106us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:53.675594 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:53.675634 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 20:28:53.676624 (+ 990us) client_negotiation.cc:240] Negotiation successful
0929 20:28:53.676792 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3197,"thread_start_us":2487,"threads_started":1}
I20260929 20:28:53.678108 31211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2d36624d754de383ad2277b852b176" candidate_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29c658233fab44e0b93c69bfef14da71" is_pre_election: true
I20260929 20:28:53.678399 31211 raft_consensus.cc:2454] T 9b2d36624d754de383ad2277b852b176 P 29c658233fab44e0b93c69bfef14da71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c4c758e48154c1aa66eaa01f5cf5eb3 in term 0.
I20260929 20:28:53.679025 31002 leader_election.cc:302] T 9b2d36624d754de383ad2277b852b176 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29c658233fab44e0b93c69bfef14da71, 5c4c758e48154c1aa66eaa01f5cf5eb3; no voters:
I20260929 20:28:53.679580 31289 raft_consensus.cc:2781] T 9b2d36624d754de383ad2277b852b176 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:28:53.679720 31289 raft_consensus.cc:482] T 9b2d36624d754de383ad2277b852b176 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:28:53.679853 31289 raft_consensus.cc:3037] T 9b2d36624d754de383ad2277b852b176 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:28:53.683384 31289 raft_consensus.cc:504] T 9b2d36624d754de383ad2277b852b176 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 } } peers { permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 } } peers { permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 } }
I20260929 20:28:53.684240 31289 leader_election.cc:288] T 9b2d36624d754de383ad2277b852b176 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [CANDIDATE]: Term 1 election: Requested vote from peers ee58c80790c7457cbae8da3c197e1a8b (127.28.244.129:34459), 29c658233fab44e0b93c69bfef14da71 (127.28.244.131:44105)
I20260929 20:28:53.684909 31211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2d36624d754de383ad2277b852b176" candidate_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29c658233fab44e0b93c69bfef14da71"
I20260929 20:28:53.685163 31211 raft_consensus.cc:3037] T 9b2d36624d754de383ad2277b852b176 P 29c658233fab44e0b93c69bfef14da71 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:28:53.686569 31150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:53.673208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:34459 (local address 127.28.244.131:48863)
0929 20:28:53.674282 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:53.674302 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:53.674425 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:53.675923 (+ 1498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:53.675931 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:53.675969 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:28:53.676232 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:53.676246 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.679947 (+ 3701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.679956 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:53.682474 (+ 2518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:53.682499 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.685529 (+ 3030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.685539 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:53.685797 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:53.685863 (+ 66us) client_negotiation.cc:769] Sending connection context
0929 20:28:53.685954 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 20:28:53.686143 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":932}
I20260929 20:28:53.693374 31277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:53.674835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:48863 (local address 127.28.244.129:34459)
0929 20:28:53.675533 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:53.675541 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:53.675564 (+ 23us) server_negotiation.cc:407] Connection header received
0929 20:28:53.675633 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:53.675641 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:53.675734 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:53.676387 (+ 653us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:28:53.676757 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.679730 (+ 2973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.682990 (+ 3260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.685293 (+ 2303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.692540 (+ 7247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:53.692667 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:53.692735 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 20:28:53.692950 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"threads_started":1}
I20260929 20:28:53.696933 31275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:53.682547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:47181 (local address 127.28.244.130:34117)
0929 20:28:53.682741 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:53.682750 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:53.687099 (+ 4349us) server_negotiation.cc:407] Connection header received
0929 20:28:53.687390 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:53.687398 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:53.687505 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:53.687574 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:28:53.688249 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.689652 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.693659 (+ 4007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.695430 (+ 1771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.695620 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:53.695730 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:53.696417 (+ 687us) server_negotiation.cc:299] Negotiation successful
0929 20:28:53.696580 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 20:28:53.697829 31211 raft_consensus.cc:2454] T 9b2d36624d754de383ad2277b852b176 P 29c658233fab44e0b93c69bfef14da71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c4c758e48154c1aa66eaa01f5cf5eb3 in term 1.
I20260929 20:28:53.698513 31002 leader_election.cc:302] T 9b2d36624d754de383ad2277b852b176 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29c658233fab44e0b93c69bfef14da71, 5c4c758e48154c1aa66eaa01f5cf5eb3; no voters:
I20260929 20:28:53.698987 31289 raft_consensus.cc:2781] T 9b2d36624d754de383ad2277b852b176 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:28:53.700435 31311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:53.664496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:34459 (local address 127.28.244.130:45661)
0929 20:28:53.667389 (+ 2893us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:53.667415 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:53.667660 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:53.668143 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:53.668150 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:53.668184 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:28:53.668560 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:53.668575 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.669878 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.669884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:53.672099 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:53.672114 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.674054 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.674059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:53.674239 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:53.674275 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 20:28:53.697984 (+ 23709us) client_negotiation.cc:240] Negotiation successful
0929 20:28:53.699991 (+ 2007us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2714,"mutex_wait_us":1,"thread_start_us":3004,"threads_started":1}
I20260929 20:28:53.697902 31150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:53.673728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:34117 (local address 127.28.244.131:47181)
0929 20:28:53.687003 (+ 13275us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:53.687025 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:53.687220 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:53.687819 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:53.687827 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:53.687861 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:28:53.688081 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:53.688090 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.689892 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.689901 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:53.692211 (+ 2310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:53.692232 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.695971 (+ 3739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.695981 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:53.696224 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:53.696258 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 20:28:53.697233 (+ 975us) client_negotiation.cc:240] Negotiation successful
0929 20:28:53.697452 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13132,"threads_started":1}
I20260929 20:28:53.701450 30932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2d36624d754de383ad2277b852b176" candidate_uuid: "29c658233fab44e0b93c69bfef14da71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee58c80790c7457cbae8da3c197e1a8b" is_pre_election: true
I20260929 20:28:53.701956 30932 raft_consensus.cc:2454] T 9b2d36624d754de383ad2277b852b176 P ee58c80790c7457cbae8da3c197e1a8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29c658233fab44e0b93c69bfef14da71 in term 0.
I20260929 20:28:53.702693 30932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2d36624d754de383ad2277b852b176" candidate_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee58c80790c7457cbae8da3c197e1a8b"
I20260929 20:28:53.702598 30931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2d36624d754de383ad2277b852b176" candidate_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee58c80790c7457cbae8da3c197e1a8b" is_pre_election: true
I20260929 20:28:53.702865 30932 raft_consensus.cc:3037] T 9b2d36624d754de383ad2277b852b176 P ee58c80790c7457cbae8da3c197e1a8b [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:28:53.703927 31145 leader_election.cc:302] T 9b2d36624d754de383ad2277b852b176 P 29c658233fab44e0b93c69bfef14da71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29c658233fab44e0b93c69bfef14da71, ee58c80790c7457cbae8da3c197e1a8b; no voters:
I20260929 20:28:53.704377 31287 raft_consensus.cc:2741] T 9b2d36624d754de383ad2277b852b176 P 29c658233fab44e0b93c69bfef14da71 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260929 20:28:53.705178 31076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2d36624d754de383ad2277b852b176" candidate_uuid: "29c658233fab44e0b93c69bfef14da71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" is_pre_election: true
I20260929 20:28:53.706624 30932 raft_consensus.cc:2454] T 9b2d36624d754de383ad2277b852b176 P ee58c80790c7457cbae8da3c197e1a8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c4c758e48154c1aa66eaa01f5cf5eb3 in term 1.
I20260929 20:28:53.707719 31289 raft_consensus.cc:686] T 9b2d36624d754de383ad2277b852b176 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [term 1 LEADER]: Becoming Leader. State: Replica: 5c4c758e48154c1aa66eaa01f5cf5eb3, State: Running, Role: LEADER
I20260929 20:28:53.708233 31289 consensus_queue.cc:237] T 9b2d36624d754de383ad2277b852b176 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 } } peers { permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 } } peers { permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 } }
I20260929 20:28:53.712289 30774 catalog_manager.cc:5495] T 9b2d36624d754de383ad2277b852b176 P 5c4c758e48154c1aa66eaa01f5cf5eb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c4c758e48154c1aa66eaa01f5cf5eb3 (127.28.244.130). New cstate: current_term: 1 leader_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 } health_report { overall_health: UNKNOWN } } }
Sep 29 20:28:53 dist-test-slave-p29l krb5kdc[30730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713731, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:28:53.779346 31010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:53.762717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55858 (local address 127.28.244.190:34953)
0929 20:28:53.762960 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:53.762968 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:53.763587 (+ 619us) server_negotiation.cc:407] Connection header received
0929 20:28:53.763782 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:53.763786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:53.763914 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:53.764000 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:53.765567 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.766268 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.768212 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.768687 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.768785 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:53.773003 (+ 4218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:53.773010 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:53.773015 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:53.773052 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:53.776225 (+ 3173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:53.776990 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:53.777014 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:53.777017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:53.777195 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:53.777573 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:53.777580 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:53.777583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:53.777864 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:53.778067 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:53.778567 (+ 500us) server_negotiation.cc:299] Negotiation successful
0929 20:28:53.778968 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":175,"server-negotiator.queue_time_us":73}
I20260929 20:28:53.779392 31321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:53.762747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:34953 (local address 127.0.0.1:55858)
0929 20:28:53.763463 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:53.763484 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:53.763648 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:53.764071 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:53.764078 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:53.765041 (+ 963us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:28:53.765404 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:53.765419 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.766414 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.766422 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:53.768070 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:53.768086 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.768949 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.768956 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:53.769130 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:53.769987 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:28:53.770013 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:28:53.772792 (+ 2779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:28:53.776368 (+ 3576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:53.776380 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:53.776387 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:53.776841 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:53.777311 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:53.777322 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:53.777324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:53.777448 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:53.778009 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:28:53.778019 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:28:53.778278 (+ 259us) client_negotiation.cc:769] Sending connection context
0929 20:28:53.778471 (+ 193us) client_negotiation.cc:240] Negotiation successful
0929 20:28:53.778847 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":25,"thread_start_us":370,"threads_started":1}
I20260929 20:28:53.785894 30774 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:55858:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 20:28:53.786993 30774 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:28:53.794351 30912 tablet_service.cc:1467] Processing CreateTablet for tablet 4881a583888f4173b84bd7535a194638 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=52b156cd677949ab9b7ceba73a2b5d92]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:28:53.794579 31052 tablet_service.cc:1467] Processing CreateTablet for tablet 4881a583888f4173b84bd7535a194638 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=52b156cd677949ab9b7ceba73a2b5d92]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:28:53.795042 30912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4881a583888f4173b84bd7535a194638. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:28:53.795135 31052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4881a583888f4173b84bd7535a194638. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:28:53.795089 31191 tablet_service.cc:1467] Processing CreateTablet for tablet 4881a583888f4173b84bd7535a194638 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=52b156cd677949ab9b7ceba73a2b5d92]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:28:53.795624 31191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4881a583888f4173b84bd7535a194638. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:28:53.802881 31280 tablet_bootstrap.cc:492] T 4881a583888f4173b84bd7535a194638 P ee58c80790c7457cbae8da3c197e1a8b: Bootstrap starting.
I20260929 20:28:53.802879 31281 tablet_bootstrap.cc:492] T 4881a583888f4173b84bd7535a194638 P 5c4c758e48154c1aa66eaa01f5cf5eb3: Bootstrap starting.
I20260929 20:28:53.804504 31282 tablet_bootstrap.cc:492] T 4881a583888f4173b84bd7535a194638 P 29c658233fab44e0b93c69bfef14da71: Bootstrap starting.
I20260929 20:28:53.804780 31281 tablet_bootstrap.cc:654] T 4881a583888f4173b84bd7535a194638 P 5c4c758e48154c1aa66eaa01f5cf5eb3: Neither blocks nor log segments found. Creating new log.
I20260929 20:28:53.806514 31282 tablet_bootstrap.cc:654] T 4881a583888f4173b84bd7535a194638 P 29c658233fab44e0b93c69bfef14da71: Neither blocks nor log segments found. Creating new log.
I20260929 20:28:53.809958 31281 tablet_bootstrap.cc:492] T 4881a583888f4173b84bd7535a194638 P 5c4c758e48154c1aa66eaa01f5cf5eb3: No bootstrap required, opened a new log
I20260929 20:28:53.810053 31280 tablet_bootstrap.cc:654] T 4881a583888f4173b84bd7535a194638 P ee58c80790c7457cbae8da3c197e1a8b: Neither blocks nor log segments found. Creating new log.
I20260929 20:28:53.810130 31282 tablet_bootstrap.cc:492] T 4881a583888f4173b84bd7535a194638 P 29c658233fab44e0b93c69bfef14da71: No bootstrap required, opened a new log
I20260929 20:28:53.810276 31282 ts_tablet_manager.cc:1397] T 4881a583888f4173b84bd7535a194638 P 29c658233fab44e0b93c69bfef14da71: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260929 20:28:53.810252 31281 ts_tablet_manager.cc:1397] T 4881a583888f4173b84bd7535a194638 P 5c4c758e48154c1aa66eaa01f5cf5eb3: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20260929 20:28:53.811033 31282 raft_consensus.cc:348] T 4881a583888f4173b84bd7535a194638 P 29c658233fab44e0b93c69bfef14da71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 } } peers { permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 } } peers { permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 } }
I20260929 20:28:53.811033 31281 raft_consensus.cc:348] T 4881a583888f4173b84bd7535a194638 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 } } peers { permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 } } peers { permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 } }
I20260929 20:28:53.811311 31282 raft_consensus.cc:374] T 4881a583888f4173b84bd7535a194638 P 29c658233fab44e0b93c69bfef14da71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:28:53.811393 31282 raft_consensus.cc:729] T 4881a583888f4173b84bd7535a194638 P 29c658233fab44e0b93c69bfef14da71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29c658233fab44e0b93c69bfef14da71, State: Initialized, Role: FOLLOWER
I20260929 20:28:53.811478 31281 raft_consensus.cc:374] T 4881a583888f4173b84bd7535a194638 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:28:53.811566 31281 raft_consensus.cc:729] T 4881a583888f4173b84bd7535a194638 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c4c758e48154c1aa66eaa01f5cf5eb3, State: Initialized, Role: FOLLOWER
I20260929 20:28:53.812395 31282 consensus_queue.cc:260] T 4881a583888f4173b84bd7535a194638 P 29c658233fab44e0b93c69bfef14da71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 } } peers { permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 } } peers { permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 } }
I20260929 20:28:53.813161 31287 tablet_replica.cc:439] T 4881a583888f4173b84bd7535a194638 P 29c658233fab44e0b93c69bfef14da71: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 } } peers { permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 } } peers { permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 } } }
I20260929 20:28:53.813342 31282 ts_tablet_manager.cc:1428] T 4881a583888f4173b84bd7535a194638 P 29c658233fab44e0b93c69bfef14da71: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 20:28:53.813477 31287 tablet_replica.cc:442] T 4881a583888f4173b84bd7535a194638 P 29c658233fab44e0b93c69bfef14da71: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:28:53.813452 31281 consensus_queue.cc:260] T 4881a583888f4173b84bd7535a194638 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 } } peers { permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 } } peers { permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 } }
I20260929 20:28:53.814286 31289 tablet_replica.cc:439] T 4881a583888f4173b84bd7535a194638 P 5c4c758e48154c1aa66eaa01f5cf5eb3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 } } peers { permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 } } peers { permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 } } }
I20260929 20:28:53.814610 31289 tablet_replica.cc:442] T 4881a583888f4173b84bd7535a194638 P 5c4c758e48154c1aa66eaa01f5cf5eb3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:28:53.815084 31281 ts_tablet_manager.cc:1428] T 4881a583888f4173b84bd7535a194638 P 5c4c758e48154c1aa66eaa01f5cf5eb3: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260929 20:28:53.822155 31280 tablet_bootstrap.cc:492] T 4881a583888f4173b84bd7535a194638 P ee58c80790c7457cbae8da3c197e1a8b: No bootstrap required, opened a new log
I20260929 20:28:53.822438 31280 ts_tablet_manager.cc:1397] T 4881a583888f4173b84bd7535a194638 P ee58c80790c7457cbae8da3c197e1a8b: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.004s
I20260929 20:28:53.823184 31280 raft_consensus.cc:348] T 4881a583888f4173b84bd7535a194638 P ee58c80790c7457cbae8da3c197e1a8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 } } peers { permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 } } peers { permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 } }
I20260929 20:28:53.823462 31280 raft_consensus.cc:374] T 4881a583888f4173b84bd7535a194638 P ee58c80790c7457cbae8da3c197e1a8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:28:53.823594 31280 raft_consensus.cc:729] T 4881a583888f4173b84bd7535a194638 P ee58c80790c7457cbae8da3c197e1a8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee58c80790c7457cbae8da3c197e1a8b, State: Initialized, Role: FOLLOWER
I20260929 20:28:53.823980 31280 consensus_queue.cc:260] T 4881a583888f4173b84bd7535a194638 P ee58c80790c7457cbae8da3c197e1a8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 } } peers { permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 } } peers { permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 } }
I20260929 20:28:53.824484 31286 tablet_replica.cc:439] T 4881a583888f4173b84bd7535a194638 P ee58c80790c7457cbae8da3c197e1a8b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 } } peers { permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 } } peers { permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 } } }
I20260929 20:28:53.824716 31280 ts_tablet_manager.cc:1428] T 4881a583888f4173b84bd7535a194638 P ee58c80790c7457cbae8da3c197e1a8b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 20:28:53.824790 31286 tablet_replica.cc:442] T 4881a583888f4173b84bd7535a194638 P ee58c80790c7457cbae8da3c197e1a8b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:28:53.865180 31289 raft_consensus.cc:482] T 4881a583888f4173b84bd7535a194638 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:28:53.865432 31289 raft_consensus.cc:504] T 4881a583888f4173b84bd7535a194638 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 } } peers { permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 } } peers { permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 } }
I20260929 20:28:53.866273 31289 leader_election.cc:288] T 4881a583888f4173b84bd7535a194638 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee58c80790c7457cbae8da3c197e1a8b (127.28.244.129:34459), 29c658233fab44e0b93c69bfef14da71 (127.28.244.131:44105)
I20260929 20:28:53.866858 31211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4881a583888f4173b84bd7535a194638" candidate_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29c658233fab44e0b93c69bfef14da71" is_pre_election: true
I20260929 20:28:53.866897 30932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4881a583888f4173b84bd7535a194638" candidate_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee58c80790c7457cbae8da3c197e1a8b" is_pre_election: true
I20260929 20:28:53.867153 31211 raft_consensus.cc:2454] T 4881a583888f4173b84bd7535a194638 P 29c658233fab44e0b93c69bfef14da71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c4c758e48154c1aa66eaa01f5cf5eb3 in term 0.
I20260929 20:28:53.867223 30932 raft_consensus.cc:2454] T 4881a583888f4173b84bd7535a194638 P ee58c80790c7457cbae8da3c197e1a8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c4c758e48154c1aa66eaa01f5cf5eb3 in term 0.
I20260929 20:28:53.867658 31002 leader_election.cc:302] T 4881a583888f4173b84bd7535a194638 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29c658233fab44e0b93c69bfef14da71, 5c4c758e48154c1aa66eaa01f5cf5eb3; no voters:
I20260929 20:28:53.868192 31289 raft_consensus.cc:2781] T 4881a583888f4173b84bd7535a194638 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:28:53.868312 31289 raft_consensus.cc:482] T 4881a583888f4173b84bd7535a194638 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:28:53.868458 31289 raft_consensus.cc:3037] T 4881a583888f4173b84bd7535a194638 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:28:53.870694 31289 raft_consensus.cc:504] T 4881a583888f4173b84bd7535a194638 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 } } peers { permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 } } peers { permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 } }
I20260929 20:28:53.871356 31289 leader_election.cc:288] T 4881a583888f4173b84bd7535a194638 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [CANDIDATE]: Term 1 election: Requested vote from peers ee58c80790c7457cbae8da3c197e1a8b (127.28.244.129:34459), 29c658233fab44e0b93c69bfef14da71 (127.28.244.131:44105)
I20260929 20:28:53.871939 30932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4881a583888f4173b84bd7535a194638" candidate_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee58c80790c7457cbae8da3c197e1a8b"
I20260929 20:28:53.871997 31211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4881a583888f4173b84bd7535a194638" candidate_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29c658233fab44e0b93c69bfef14da71"
I20260929 20:28:53.872218 31211 raft_consensus.cc:3037] T 4881a583888f4173b84bd7535a194638 P 29c658233fab44e0b93c69bfef14da71 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:28:53.872225 30932 raft_consensus.cc:3037] T 4881a583888f4173b84bd7535a194638 P ee58c80790c7457cbae8da3c197e1a8b [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:28:53.875382 31211 raft_consensus.cc:2454] T 4881a583888f4173b84bd7535a194638 P 29c658233fab44e0b93c69bfef14da71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c4c758e48154c1aa66eaa01f5cf5eb3 in term 1.
I20260929 20:28:53.875478 30932 raft_consensus.cc:2454] T 4881a583888f4173b84bd7535a194638 P ee58c80790c7457cbae8da3c197e1a8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c4c758e48154c1aa66eaa01f5cf5eb3 in term 1.
I20260929 20:28:53.876096 31005 leader_election.cc:302] T 4881a583888f4173b84bd7535a194638 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 29c658233fab44e0b93c69bfef14da71, 5c4c758e48154c1aa66eaa01f5cf5eb3, ee58c80790c7457cbae8da3c197e1a8b; no voters:
I20260929 20:28:53.876484 31289 raft_consensus.cc:2781] T 4881a583888f4173b84bd7535a194638 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:28:53.876677 31289 raft_consensus.cc:686] T 4881a583888f4173b84bd7535a194638 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [term 1 LEADER]: Becoming Leader. State: Replica: 5c4c758e48154c1aa66eaa01f5cf5eb3, State: Running, Role: LEADER
I20260929 20:28:53.876945 31289 consensus_queue.cc:237] T 4881a583888f4173b84bd7535a194638 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 } } peers { permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 } } peers { permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 } }
I20260929 20:28:53.879204 31314 tablet_replica.cc:439] T 4881a583888f4173b84bd7535a194638 P 5c4c758e48154c1aa66eaa01f5cf5eb3: TxnStatusTablet state changed. Reason: New leader 5c4c758e48154c1aa66eaa01f5cf5eb3. Latest consensus state: current_term: 1 leader_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 } } peers { permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 } } peers { permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 } } }
I20260929 20:28:53.879695 31314 tablet_replica.cc:442] T 4881a583888f4173b84bd7535a194638 P 5c4c758e48154c1aa66eaa01f5cf5eb3: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:28:53.880491 31326 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 20:28:53.880751 31326 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 20:28:53.880923 30775 catalog_manager.cc:5495] T 4881a583888f4173b84bd7535a194638 P 5c4c758e48154c1aa66eaa01f5cf5eb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c4c758e48154c1aa66eaa01f5cf5eb3 (127.28.244.130). New cstate: current_term: 1 leader_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 } health_report { overall_health: UNKNOWN } } }
I20260929 20:28:53.911849 31275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:53.903530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44334 (local address 127.28.244.130:34117)
0929 20:28:53.903751 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:53.903758 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:53.904148 (+ 390us) server_negotiation.cc:407] Connection header received
0929 20:28:53.904490 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:53.904504 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:53.904624 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:53.904799 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:28:53.905305 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.906553 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.908628 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.909315 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.909463 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:53.910134 (+ 671us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:28:53.911046 (+ 912us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:28:53.911219 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:53.911367 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 20:28:53.911478 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 20:28:53.912150 31321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:53.903464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:34117 (local address 127.0.0.1:44334)
0929 20:28:53.904017 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:53.904038 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:53.904232 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:53.904835 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:53.904841 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:53.904854 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:28:53.905138 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:53.905151 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.906727 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.906733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:53.908432 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:53.908453 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.909523 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.909531 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:53.909782 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:53.909882 (+ 100us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:28:53.911189 (+ 1307us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:28:53.911219 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 20:28:53.911321 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 20:28:53.911580 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318}
I20260929 20:28:53.917188 31211 raft_consensus.cc:1261] T 4881a583888f4173b84bd7535a194638 P 29c658233fab44e0b93c69bfef14da71 [term 1 FOLLOWER]: Refusing update from remote peer 5c4c758e48154c1aa66eaa01f5cf5eb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:28:53.917656 30932 raft_consensus.cc:1261] T 4881a583888f4173b84bd7535a194638 P ee58c80790c7457cbae8da3c197e1a8b [term 1 FOLLOWER]: Refusing update from remote peer 5c4c758e48154c1aa66eaa01f5cf5eb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:28:53.917951 31314 consensus_queue.cc:1035] T 4881a583888f4173b84bd7535a194638 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:28:53.918457 31289 consensus_queue.cc:1035] T 4881a583888f4173b84bd7535a194638 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:28:53.932855 31287 tablet_replica.cc:439] T 4881a583888f4173b84bd7535a194638 P 29c658233fab44e0b93c69bfef14da71: TxnStatusTablet state changed. Reason: New leader 5c4c758e48154c1aa66eaa01f5cf5eb3. Latest consensus state: current_term: 1 leader_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 } } peers { permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 } } peers { permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 } } }
I20260929 20:28:53.933238 31287 tablet_replica.cc:442] T 4881a583888f4173b84bd7535a194638 P 29c658233fab44e0b93c69bfef14da71: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:28:53.937201 31289 tablet_replica.cc:439] T 4881a583888f4173b84bd7535a194638 P 5c4c758e48154c1aa66eaa01f5cf5eb3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 } } peers { permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 } } peers { permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 } } }
I20260929 20:28:53.937588 31289 tablet_replica.cc:442] T 4881a583888f4173b84bd7535a194638 P 5c4c758e48154c1aa66eaa01f5cf5eb3: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:28:53.938598 31287 tablet_replica.cc:439] T 4881a583888f4173b84bd7535a194638 P 29c658233fab44e0b93c69bfef14da71: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 } } peers { permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 } } peers { permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 } } }
I20260929 20:28:53.938867 31287 tablet_replica.cc:442] T 4881a583888f4173b84bd7535a194638 P 29c658233fab44e0b93c69bfef14da71: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:28:53.940138 31286 tablet_replica.cc:439] T 4881a583888f4173b84bd7535a194638 P ee58c80790c7457cbae8da3c197e1a8b: TxnStatusTablet state changed. Reason: New leader 5c4c758e48154c1aa66eaa01f5cf5eb3. Latest consensus state: current_term: 1 leader_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 } } peers { permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 } } peers { permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 } } }
I20260929 20:28:53.940454 31286 tablet_replica.cc:442] T 4881a583888f4173b84bd7535a194638 P ee58c80790c7457cbae8da3c197e1a8b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:28:53.945236 31289 tablet_replica.cc:439] T 4881a583888f4173b84bd7535a194638 P 5c4c758e48154c1aa66eaa01f5cf5eb3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 } } peers { permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 } } peers { permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 } } }
I20260929 20:28:53.945549 31289 tablet_replica.cc:442] T 4881a583888f4173b84bd7535a194638 P 5c4c758e48154c1aa66eaa01f5cf5eb3: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:28:53.948879 31286 tablet_replica.cc:439] T 4881a583888f4173b84bd7535a194638 P ee58c80790c7457cbae8da3c197e1a8b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 } } peers { permanent_uuid: "5c4c758e48154c1aa66eaa01f5cf5eb3" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34117 } } peers { permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 } } }
I20260929 20:28:53.949173 31286 tablet_replica.cc:442] T 4881a583888f4173b84bd7535a194638 P ee58c80790c7457cbae8da3c197e1a8b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:28:53.960302 31275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:53.952813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44344 (local address 127.28.244.130:34117)
0929 20:28:53.953413 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:53.953423 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:53.953442 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:28:53.953528 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:53.953535 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:53.953635 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:53.953742 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:28:53.954382 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.955581 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.957709 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.958502 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.958644 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:53.959274 (+ 630us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:28:53.959434 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:28:53.959541 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:53.959761 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 20:28:53.959903 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471}
I20260929 20:28:53.982292 31275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:53.973987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:55631 (local address 127.28.244.130:34117)
0929 20:28:53.974193 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:53.974200 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:53.974941 (+ 741us) server_negotiation.cc:407] Connection header received
0929 20:28:53.975251 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:53.975257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:53.975348 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:53.975460 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:28:53.976229 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.977411 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.979899 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.980668 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.980803 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:53.981382 (+ 579us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:28:53.981568 (+ 186us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:28:53.981682 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:53.981849 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 20:28:53.981956 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":74}
I20260929 20:28:53.982430 31333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:53.974059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:34117 (local address 127.28.244.130:55631)
0929 20:28:53.974832 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:53.974855 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:53.975070 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:53.975488 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:53.975494 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:53.975507 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:28:53.976016 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:53.976029 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.977557 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.977564 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:53.979693 (+ 2129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:53.979712 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:53.980856 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:53.980863 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:53.981062 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:53.981155 (+ 93us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:28:53.981700 (+ 545us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:28:53.981712 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 20:28:53.981801 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 20:28:53.981968 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":3,"thread_start_us":424,"threads_started":1}
I20260929 20:28:53.996954 30932 raft_consensus.cc:1261] T 9b2d36624d754de383ad2277b852b176 P ee58c80790c7457cbae8da3c197e1a8b [term 1 FOLLOWER]: Refusing update from remote peer 5c4c758e48154c1aa66eaa01f5cf5eb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:28:53.996968 31210 raft_consensus.cc:1261] T 9b2d36624d754de383ad2277b852b176 P 29c658233fab44e0b93c69bfef14da71 [term 1 FOLLOWER]: Refusing update from remote peer 5c4c758e48154c1aa66eaa01f5cf5eb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:28:53.997663 31289 consensus_queue.cc:1035] T 9b2d36624d754de383ad2277b852b176 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee58c80790c7457cbae8da3c197e1a8b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:28:53.997972 31314 consensus_queue.cc:1035] T 9b2d36624d754de383ad2277b852b176 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29c658233fab44e0b93c69bfef14da71" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:28:54.015431 31295 mvcc.cc:205] Tried to move back new op lower bound from 7334763454445252608 to 7334763453273305088. Current Snapshot: MvccSnapshot[applied={T|T < 7334763454445252608}]
I20260929 20:28:54.053006 31294 mvcc.cc:205] Tried to move back new op lower bound from 7334763454445252608 to 7334763453273305088. Current Snapshot: MvccSnapshot[applied={T|T < 7334763454445252608}]
I20260929 20:28:54.117563 30773 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:28:54.117847 30773 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:28:54.122148 30773 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a039d1b8e4f4eb9849b1bf235f81620: Sending DeleteTablet for 3 replicas of tablet 9b2d36624d754de383ad2277b852b176
I20260929 20:28:54.123245 30912 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b2d36624d754de383ad2277b852b176 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:28:54 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:43578
I20260929 20:28:54.123958 31191 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b2d36624d754de383ad2277b852b176 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:28:54 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:49634
I20260929 20:28:54.123970 31051 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b2d36624d754de383ad2277b852b176 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:28:54 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:44320
I20260929 20:28:54.136385 31339 tablet_replica.cc:330] T 9b2d36624d754de383ad2277b852b176 P 29c658233fab44e0b93c69bfef14da71: stopping tablet replica
I20260929 20:28:54.136786 31339 raft_consensus.cc:2229] T 9b2d36624d754de383ad2277b852b176 P 29c658233fab44e0b93c69bfef14da71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:28:54.137125 31339 raft_consensus.cc:2258] T 9b2d36624d754de383ad2277b852b176 P 29c658233fab44e0b93c69bfef14da71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:28:54.138386 31339 ts_tablet_manager.cc:1905] T 9b2d36624d754de383ad2277b852b176 P 29c658233fab44e0b93c69bfef14da71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:28:54.144677 31340 tablet_replica.cc:330] T 9b2d36624d754de383ad2277b852b176 P 5c4c758e48154c1aa66eaa01f5cf5eb3: stopping tablet replica
I20260929 20:28:54.144955 31340 raft_consensus.cc:2229] T 9b2d36624d754de383ad2277b852b176 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:28:54.145982 31338 tablet_replica.cc:330] T 9b2d36624d754de383ad2277b852b176 P ee58c80790c7457cbae8da3c197e1a8b: stopping tablet replica
I20260929 20:28:54.146282 31338 raft_consensus.cc:2229] T 9b2d36624d754de383ad2277b852b176 P ee58c80790c7457cbae8da3c197e1a8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:28:54.146307 31339 ts_tablet_manager.cc:1918] T 9b2d36624d754de383ad2277b852b176 P 29c658233fab44e0b93c69bfef14da71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:28:54.146546 31339 log.cc:1196] T 9b2d36624d754de383ad2277b852b176 P 29c658233fab44e0b93c69bfef14da71: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-2/wal/wals/9b2d36624d754de383ad2277b852b176
I20260929 20:28:54.146570 31338 raft_consensus.cc:2258] T 9b2d36624d754de383ad2277b852b176 P ee58c80790c7457cbae8da3c197e1a8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:28:54.146855 31340 raft_consensus.cc:2258] T 9b2d36624d754de383ad2277b852b176 P 5c4c758e48154c1aa66eaa01f5cf5eb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:28:54.147511 31339 ts_tablet_manager.cc:1939] T 9b2d36624d754de383ad2277b852b176 P 29c658233fab44e0b93c69bfef14da71: Deleting consensus metadata
I20260929 20:28:54.148479 31338 ts_tablet_manager.cc:1905] T 9b2d36624d754de383ad2277b852b176 P ee58c80790c7457cbae8da3c197e1a8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:28:54.149155 31340 ts_tablet_manager.cc:1905] T 9b2d36624d754de383ad2277b852b176 P 5c4c758e48154c1aa66eaa01f5cf5eb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:28:54.149776 30759 catalog_manager.cc:4841] TS 29c658233fab44e0b93c69bfef14da71 (127.28.244.131:44105): tablet 9b2d36624d754de383ad2277b852b176 (table test-table [id=bb592b8eeaec43b591866e8cda7a1c8a]) successfully deleted
I20260929 20:28:54.156783 31338 ts_tablet_manager.cc:1918] T 9b2d36624d754de383ad2277b852b176 P ee58c80790c7457cbae8da3c197e1a8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:28:54.157303 31338 log.cc:1196] T 9b2d36624d754de383ad2277b852b176 P ee58c80790c7457cbae8da3c197e1a8b: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-0/wal/wals/9b2d36624d754de383ad2277b852b176
I20260929 20:28:54.157902 31338 ts_tablet_manager.cc:1939] T 9b2d36624d754de383ad2277b852b176 P ee58c80790c7457cbae8da3c197e1a8b: Deleting consensus metadata
I20260929 20:28:54.158138 31340 ts_tablet_manager.cc:1918] T 9b2d36624d754de383ad2277b852b176 P 5c4c758e48154c1aa66eaa01f5cf5eb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:28:54.158337 31340 log.cc:1196] T 9b2d36624d754de383ad2277b852b176 P 5c4c758e48154c1aa66eaa01f5cf5eb3: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713721201454-29650-0/minicluster-data/ts-1/wal/wals/9b2d36624d754de383ad2277b852b176
I20260929 20:28:54.158870 31340 ts_tablet_manager.cc:1939] T 9b2d36624d754de383ad2277b852b176 P 5c4c758e48154c1aa66eaa01f5cf5eb3: Deleting consensus metadata
I20260929 20:28:54.159129 30762 catalog_manager.cc:4841] TS ee58c80790c7457cbae8da3c197e1a8b (127.28.244.129:34459): tablet 9b2d36624d754de383ad2277b852b176 (table test-table [id=bb592b8eeaec43b591866e8cda7a1c8a]) successfully deleted
I20260929 20:28:54.160081 30759 catalog_manager.cc:4841] TS 5c4c758e48154c1aa66eaa01f5cf5eb3 (127.28.244.130:34117): tablet 9b2d36624d754de383ad2277b852b176 (table test-table [id=bb592b8eeaec43b591866e8cda7a1c8a]) successfully deleted
Sep 29 20:28:54 dist-test-slave-p29l krb5kdc[30730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713733, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.244.129@KRBTEST.COM
I20260929 20:28:54.172399 31313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:54.151702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43594 (local address 127.28.244.129:34459)
0929 20:28:54.151967 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:54.151974 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:54.154395 (+ 2421us) server_negotiation.cc:407] Connection header received
0929 20:28:54.154780 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:54.154786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:54.154878 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:54.154957 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:54.157311 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:54.158553 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:54.160573 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:54.161133 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:54.161282 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:54.166048 (+ 4766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:54.166064 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:54.166080 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:54.166122 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:54.169359 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:54.170327 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:54.170335 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:54.170337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:54.170474 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:54.170883 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:54.170886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:54.170888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:54.171221 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:54.171454 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:54.171867 (+ 413us) server_negotiation.cc:299] Negotiation successful
0929 20:28:54.172072 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 20:28:54.173216 30977 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43594
I20260929 20:28:54.190768 31010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:54.176746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55866 (local address 127.28.244.190:34953)
0929 20:28:54.176934 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:54.176940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:54.177661 (+ 721us) server_negotiation.cc:407] Connection header received
0929 20:28:54.177869 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:54.177875 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:54.177956 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:54.178067 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:54.179572 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:54.180521 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:54.181797 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:54.182226 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:54.182371 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:54.184902 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:54.184916 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:54.184921 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:54.184959 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:54.187865 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:54.188652 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:54.188659 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:54.188663 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:54.188796 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:54.189186 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:54.189192 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:54.189195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:54.189557 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:54.189837 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:54.190169 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 20:28:54.190408 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 20:28:54.191362 30773 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55866
I20260929 20:28:54.194376 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 30846
I20260929 20:28:54.209112 31313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:54.195274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43602 (local address 127.28.244.129:34459)
0929 20:28:54.195641 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:54.195647 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:54.196092 (+ 445us) server_negotiation.cc:407] Connection header received
0929 20:28:54.196284 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:54.196290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:54.196371 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:54.196474 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:54.197636 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:54.198332 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:54.199601 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:54.200208 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:54.200340 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:54.202896 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:54.202903 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:54.202909 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:54.202956 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:54.206128 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:54.207009 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:54.207018 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:54.207022 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:54.207168 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:54.207576 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:54.207582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:54.207585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:54.207936 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:54.208223 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:54.208554 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 20:28:54.208794 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219}
W20260929 20:28:54.209692 30977 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43602
W20260929 20:28:54.210330 29650 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 30846: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:28:54.210548 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 30846
I20260929 20:28:54.242759 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 30986
Sep 29 20:28:54 dist-test-slave-p29l krb5kdc[30730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713733, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.244.130@KRBTEST.COM
I20260929 20:28:54.263530 31275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:54.243590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44346 (local address 127.28.244.130:34117)
0929 20:28:54.243788 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:54.243796 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:54.243841 (+ 45us) server_negotiation.cc:407] Connection header received
0929 20:28:54.244061 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:54.244067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:54.244146 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:54.244248 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:54.245917 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:54.247144 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:54.249177 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:54.249879 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:54.250009 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:54.255318 (+ 5309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:54.255334 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:54.255351 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:54.255400 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:54.259900 (+ 4500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:54.260874 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:54.260886 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:54.260890 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:54.261077 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:54.261572 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:54.261578 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:54.261581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:54.262178 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:54.262532 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:54.262859 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 20:28:54.263123 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 20:28:54.264432 31117 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44346
W20260929 20:28:54.265259 29650 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 30986: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:28:54.265465 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 30986
I20260929 20:28:54.286795 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 31126
Sep 29 20:28:54 dist-test-slave-p29l krb5kdc[30730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713733, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.244.131@KRBTEST.COM
I20260929 20:28:54.301692 31346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:54.287649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49636 (local address 127.28.244.131:44105)
0929 20:28:54.288188 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:54.288193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:54.288210 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:28:54.288266 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:54.288269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:54.288333 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:54.288426 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:54.289771 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:54.290526 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:54.291718 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:54.292178 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:54.292286 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:54.296314 (+ 4028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:54.296329 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:54.296346 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:54.296377 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:54.299196 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:54.299907 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:54.299916 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:54.299918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:54.300041 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:54.300370 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:54.300373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:54.300375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:54.300710 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:54.300916 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:54.301148 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 20:28:54.301315 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":320,"threads_started":1}
W20260929 20:28:54.302456 31256 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49636
W20260929 20:28:54.303107 29650 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 31126: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:28:54.303233 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 31126
I20260929 20:28:54.320515 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 30743
I20260929 20:28:54.321753 30815 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:28:54.441342 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 30743
2026-09-29T20:28:54Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4854 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:28:54 dist-test-slave-p29l krb5kdc[31350](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:28:54 dist-test-slave-p29l krb5kdc[31350](info): set up 2 sockets
Sep 29 20:28:54 dist-test-slave-p29l krb5kdc[31350](info): commencing operation
krb5kdc: starting...
W20260929 20:28:55.547571 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.005s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:28:55 dist-test-slave-p29l krb5kdc[31350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:28:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:28:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.244.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
I20260929 20:28:55.745371 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:37129
--webserver_interface=127.28.244.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:45291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:37129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:28:56.080207 31363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:28:56.080549 31363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:28:56.080732 31363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:28:56.088145 31363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:28:56.088270 31363 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 20:28:56.088349 31363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:28:56.088449 31363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:28:56.088515 31363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:28:56.088603 31363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:28:56.098896 31363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:37129
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:37129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:28:56.100394 31363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:28:56.101858 31363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:28:56.114565 31368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:56.114684 31371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:56.114600 31369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:28:56.116212 31363 server_base.cc:1034] running on GCE node
I20260929 20:28:56.117127 31363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:28:56.118716 31363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:28:56.119942 31363 hybrid_clock.cc:648] HybridClock initialized: now 1790713736119864 us; error 54 us; skew 500 ppm
I20260929 20:28:56.120307 31363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:28:56 dist-test-slave-p29l krb5kdc[31350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713736, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:28:56.123777 31363 init.cc:377] Logged in from keytab as kudu/127.28.244.190@KRBTEST.COM (short username kudu)
I20260929 20:28:56.125519 31363 webserver.cc:460] Webserver started at http://127.28.244.190:36461/ using document root <none> and password file <none>
I20260929 20:28:56.126281 31363 fs_manager.cc:359] Metadata directory not provided
I20260929 20:28:56.126425 31363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:28:56.126756 31363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:28:56.129977 31363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "5884f7b08d4841ba82729ea2581cdd91"
format_stamp: "Formatted at 2026-09-29 20:28:56 on dist-test-slave-p29l"
server_key: "0d30211f7d91b516b9b54c13fd6451b3"
server_key_iv: "81562b4c2ee5b6b00eaa7628a650e579"
server_key_version: "encryptionkey@0"
I20260929 20:28:56.130823 31363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "5884f7b08d4841ba82729ea2581cdd91"
format_stamp: "Formatted at 2026-09-29 20:28:56 on dist-test-slave-p29l"
server_key: "0d30211f7d91b516b9b54c13fd6451b3"
server_key_iv: "81562b4c2ee5b6b00eaa7628a650e579"
server_key_version: "encryptionkey@0"
I20260929 20:28:56.136479 31363 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 20:28:56.140411 31378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:56.141896 31363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 20:28:56.142097 31363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "5884f7b08d4841ba82729ea2581cdd91"
format_stamp: "Formatted at 2026-09-29 20:28:56 on dist-test-slave-p29l"
server_key: "0d30211f7d91b516b9b54c13fd6451b3"
server_key_iv: "81562b4c2ee5b6b00eaa7628a650e579"
server_key_version: "encryptionkey@0"
I20260929 20:28:56.142336 31363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:28:56.168195 31363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:28:56.173727 31363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:28:56.173965 31363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:28:56.198837 31363 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:37129
I20260929 20:28:56.198848 31439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:37129 every 8 connection(s)
I20260929 20:28:56.200376 31363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:28:56.204631 31440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:28:56.205350 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 31363
I20260929 20:28:56.205559 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:28:56.206207 29650 external_mini_cluster.cc:1428] Setting key 271a0b3557bb9f3c939f6639d74e7b99
I20260929 20:28:56.221733 31440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5884f7b08d4841ba82729ea2581cdd91: Bootstrap starting.
Sep 29 20:28:56 dist-test-slave-p29l krb5kdc[31350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:28:56.226097 31440 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5884f7b08d4841ba82729ea2581cdd91: Neither blocks nor log segments found. Creating new log.
I20260929 20:28:56.228003 31440 log.cc:824] T 00000000000000000000000000000000 P 5884f7b08d4841ba82729ea2581cdd91: Log is configured to *not* fsync() on all Append() calls
I20260929 20:28:56.231979 31440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5884f7b08d4841ba82729ea2581cdd91: No bootstrap required, opened a new log
I20260929 20:28:56.235313 31443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:56.208490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57678 (local address 127.28.244.190:37129)
0929 20:28:56.209487 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:56.209505 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:56.209563 (+ 58us) server_negotiation.cc:407] Connection header received
0929 20:28:56.212414 (+ 2851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:56.212455 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:56.213178 (+ 723us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:56.213752 (+ 574us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:56.215372 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:56.217727 (+ 2355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:56.219133 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:56.220395 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:56.220617 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:56.225251 (+ 4634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:56.225263 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:56.225287 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:56.225346 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:56.229745 (+ 4399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:56.230661 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:56.230671 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:56.230681 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:56.230977 (+ 296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:56.231447 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:56.231452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:56.231456 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:56.232346 (+ 890us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:56.232684 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:56.233169 (+ 485us) server_negotiation.cc:299] Negotiation successful
0929 20:28:56.233683 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":412,"threads_started":1}
I20260929 20:28:56.242697 31440 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5884f7b08d4841ba82729ea2581cdd91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5884f7b08d4841ba82729ea2581cdd91" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 37129 } }
I20260929 20:28:56.243104 31440 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5884f7b08d4841ba82729ea2581cdd91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:28:56.243192 31440 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5884f7b08d4841ba82729ea2581cdd91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5884f7b08d4841ba82729ea2581cdd91, State: Initialized, Role: FOLLOWER
I20260929 20:28:56.243894 31440 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5884f7b08d4841ba82729ea2581cdd91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5884f7b08d4841ba82729ea2581cdd91" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 37129 } }
I20260929 20:28:56.244136 31440 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5884f7b08d4841ba82729ea2581cdd91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:28:56.244248 31440 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5884f7b08d4841ba82729ea2581cdd91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:28:56.244436 31440 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5884f7b08d4841ba82729ea2581cdd91 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:28:56.247483 31440 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5884f7b08d4841ba82729ea2581cdd91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5884f7b08d4841ba82729ea2581cdd91" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 37129 } }
I20260929 20:28:56.248135 31440 leader_election.cc:302] T 00000000000000000000000000000000 P 5884f7b08d4841ba82729ea2581cdd91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5884f7b08d4841ba82729ea2581cdd91; no voters:
I20260929 20:28:56.248715 31440 leader_election.cc:288] T 00000000000000000000000000000000 P 5884f7b08d4841ba82729ea2581cdd91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:28:56.248873 31445 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5884f7b08d4841ba82729ea2581cdd91 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:28:56.249655 31445 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5884f7b08d4841ba82729ea2581cdd91 [term 1 LEADER]: Becoming Leader. State: Replica: 5884f7b08d4841ba82729ea2581cdd91, State: Running, Role: LEADER
I20260929 20:28:56.250337 31445 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5884f7b08d4841ba82729ea2581cdd91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5884f7b08d4841ba82729ea2581cdd91" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 37129 } }
I20260929 20:28:56.251044 31440 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5884f7b08d4841ba82729ea2581cdd91 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:28:56.253923 31447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5884f7b08d4841ba82729ea2581cdd91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5884f7b08d4841ba82729ea2581cdd91. Latest consensus state: current_term: 1 leader_uuid: "5884f7b08d4841ba82729ea2581cdd91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5884f7b08d4841ba82729ea2581cdd91" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 37129 } } }
I20260929 20:28:56.254309 31447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5884f7b08d4841ba82729ea2581cdd91 [sys.catalog]: This master's current role is: LEADER
I20260929 20:28:56.254788 31446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5884f7b08d4841ba82729ea2581cdd91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5884f7b08d4841ba82729ea2581cdd91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5884f7b08d4841ba82729ea2581cdd91" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 37129 } } }
I20260929 20:28:56.255031 31446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5884f7b08d4841ba82729ea2581cdd91 [sys.catalog]: This master's current role is: LEADER
I20260929 20:28:56.255257 31453 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:28:56.263149 31453 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:28:56.274116 31453 catalog_manager.cc:1347] Generated new cluster ID: 5ff34b3b18d046d0be452cae1f06f41b
I20260929 20:28:56.274281 31453 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:28:56.302145 31453 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:28:56.304131 31453 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:28:56.313581 31453 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5884f7b08d4841ba82729ea2581cdd91: Generated new TSK 0
I20260929 20:28:56.314500 31453 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.244.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab.
Entry for principal kudu/127.28.244.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab.
I20260929 20:28:56.368541 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.129:0
--local_ip_for_outbound_sockets=127.28.244.129
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:37129
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:45291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:28:56.719748 31466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:28:56.720178 31466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:28:56.720351 31466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:28:56.728017 31466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:28:56.728183 31466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:28:56.728333 31466 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 20:28:56.728421 31466 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:28:56.728554 31466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.129
I20260929 20:28:56.739027 31466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.244.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.244.190:37129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.129
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:28:56.740681 31466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:28:56.742107 31466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:28:56.757812 31471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:56.757756 31472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:56.757719 31474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:28:56.763993 31466 server_base.cc:1034] running on GCE node
I20260929 20:28:56.764818 31466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:28:56.766042 31466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:28:56.767354 31466 hybrid_clock.cc:648] HybridClock initialized: now 1790713736767234 us; error 71 us; skew 500 ppm
I20260929 20:28:56.767735 31466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:28:56 dist-test-slave-p29l krb5kdc[31350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713736, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:28:56.771164 31466 init.cc:377] Logged in from keytab as kudu/127.28.244.129@KRBTEST.COM (short username kudu)
I20260929 20:28:56.772897 31466 webserver.cc:460] Webserver started at http://127.28.244.129:45907/ using document root <none> and password file <none>
I20260929 20:28:56.773630 31466 fs_manager.cc:359] Metadata directory not provided
I20260929 20:28:56.773738 31466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:28:56.774020 31466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:28:56.777174 31466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-0/data/instance:
uuid: "1fd552a67209408b9d25ed9b4937d293"
format_stamp: "Formatted at 2026-09-29 20:28:56 on dist-test-slave-p29l"
server_key: "3da36bbb1f71c17b590ccd996a7b4257"
server_key_iv: "6ad777d5fb90316c2475d3b38bea44c4"
server_key_version: "encryptionkey@0"
I20260929 20:28:56.777941 31466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance:
uuid: "1fd552a67209408b9d25ed9b4937d293"
format_stamp: "Formatted at 2026-09-29 20:28:56 on dist-test-slave-p29l"
server_key: "3da36bbb1f71c17b590ccd996a7b4257"
server_key_iv: "6ad777d5fb90316c2475d3b38bea44c4"
server_key_version: "encryptionkey@0"
I20260929 20:28:56.783860 31466 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 20:28:56.787886 31481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:56.789498 31466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:28:56.789706 31466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-0/wal
uuid: "1fd552a67209408b9d25ed9b4937d293"
format_stamp: "Formatted at 2026-09-29 20:28:56 on dist-test-slave-p29l"
server_key: "3da36bbb1f71c17b590ccd996a7b4257"
server_key_iv: "6ad777d5fb90316c2475d3b38bea44c4"
server_key_version: "encryptionkey@0"
I20260929 20:28:56.789909 31466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:28:56.806071 31466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:28:56.810868 31466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:28:56.811220 31466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:28:56.814409 31466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:28:56.814538 31466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:56.814674 31466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:28:56.814778 31466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:28:56 dist-test-slave-p29l krb5kdc[31350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713736, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:28:56.838490 31490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:56.814982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:54949 (local address 127.28.244.190:37129)
0929 20:28:56.815752 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:56.815758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:56.816835 (+ 1077us) server_negotiation.cc:407] Connection header received
0929 20:28:56.819062 (+ 2227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:56.819068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:56.819161 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:56.819295 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:56.821936 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:56.822898 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:56.825641 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:56.826092 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:56.826257 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:56.831445 (+ 5188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:56.831451 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:56.831454 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:56.831490 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:56.834956 (+ 3466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:56.835883 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:56.835888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:56.835890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:56.836025 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:56.836529 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:56.836534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:56.836539 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:56.836832 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:56.837042 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:56.837862 (+ 820us) server_negotiation.cc:299] Negotiation successful
0929 20:28:56.838122 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":519,"threads_started":1}
I20260929 20:28:56.840091 31491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:56.815762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37129 (local address 127.28.244.129:54949)
0929 20:28:56.816653 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:56.816698 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:56.818734 (+ 2036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:56.819619 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:56.819638 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:56.820597 (+ 959us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:28:56.821630 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:56.821665 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:56.823091 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:56.823104 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:56.825465 (+ 2361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:56.825479 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:56.826394 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:56.826405 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:56.826822 (+ 417us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:56.827885 (+ 1063us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:28:56.827930 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:28:56.831160 (+ 3230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:28:56.835138 (+ 3978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:56.835149 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:56.835175 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:56.835688 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:56.836155 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:56.836161 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:56.836165 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:56.836392 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:56.837007 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:28:56.837016 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:28:56.837444 (+ 428us) client_negotiation.cc:769] Sending connection context
0929 20:28:56.837750 (+ 306us) client_negotiation.cc:240] Negotiation successful
0929 20:28:56.838369 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":346,"threads_started":1}
I20260929 20:28:56.866320 31553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:56.848342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37129 (local address 127.28.244.129:49129)
0929 20:28:56.849131 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:56.849157 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:56.849348 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:56.849723 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:56.849732 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:56.850583 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:28:56.851113 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:56.851133 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:56.852476 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:56.852485 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:56.854726 (+ 2241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:56.854755 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:56.855880 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:56.855890 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:56.856187 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:56.857164 (+ 977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:28:56.857198 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:28:56.858395 (+ 1197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:28:56.863103 (+ 4708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:56.863115 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:56.863121 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:56.863794 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:56.864364 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:56.864368 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:56.864370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:56.864517 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:56.865339 (+ 822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:28:56.865343 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:28:56.865549 (+ 206us) client_negotiation.cc:769] Sending connection context
0929 20:28:56.865684 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 20:28:56.865983 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":393,"threads_started":1}
I20260929 20:28:56.866734 31490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:56.848349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:49129 (local address 127.28.244.190:37129)
0929 20:28:56.848568 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:56.848576 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:56.849264 (+ 688us) server_negotiation.cc:407] Connection header received
0929 20:28:56.849484 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:56.849489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:56.849560 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:56.849648 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:56.851309 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:56.852297 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:56.855030 (+ 2733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:56.855554 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:56.855683 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:56.858677 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:56.858682 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:56.858686 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:56.858724 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:56.862901 (+ 4177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:56.863986 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:56.863996 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:56.864003 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:56.864221 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:56.864642 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:56.864650 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:56.864656 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:56.865191 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:56.865387 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:56.865854 (+ 467us) server_negotiation.cc:299] Negotiation successful
0929 20:28:56.866199 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:28:56.878818 31466 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.129:34227
I20260929 20:28:56.878859 31601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.129:34227 every 8 connection(s)
I20260929 20:28:56.880370 31466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
I20260929 20:28:56.881475 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 31466
I20260929 20:28:56.881687 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance
I20260929 20:28:56.882386 29650 external_mini_cluster.cc:1428] Setting key 17894191355beb517326e7b34051687d
I20260929 20:28:56.882910 31602 heartbeater.cc:347] Connected to a master server at 127.28.244.190:37129
I20260929 20:28:56.883172 31602 heartbeater.cc:464] Registering TS with master...
I20260929 20:28:56.884035 31602 heartbeater.cc:511] Master 127.28.244.190:37129 requested a full tablet report, sending...
I20260929 20:28:56.886598 31395 ts_manager.cc:194] Registered new tserver with Master: 1fd552a67209408b9d25ed9b4937d293 (127.28.244.129:34227)
I20260929 20:28:56.889026 31395 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.129@KRBTEST.COM'} at 127.28.244.129:54949
WARNING: no policy specified for kudu/127.28.244.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab.
Entry for principal kudu/127.28.244.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab.
I20260929 20:28:56.927589 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.130:0
--local_ip_for_outbound_sockets=127.28.244.130
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:37129
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:45291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:28:57.280179 31606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:28:57.280517 31606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:28:57.280750 31606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:28:57.289247 31606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:28:57.289378 31606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:28:57.289578 31606 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 20:28:57.289678 31606 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:28:57.289820 31606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.130
I20260929 20:28:57.301728 31606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.244.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.244.190:37129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.130
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:28:57.303191 31606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:28:57.304666 31606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:28:57.320132 31611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:57.320111 31614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:57.320169 31612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:28:57.321508 31606 server_base.cc:1034] running on GCE node
I20260929 20:28:57.322197 31606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:28:57.323264 31606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:28:57.324517 31606 hybrid_clock.cc:648] HybridClock initialized: now 1790713737324469 us; error 36 us; skew 500 ppm
I20260929 20:28:57.324849 31606 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:28:57 dist-test-slave-p29l krb5kdc[31350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713737, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:28:57.332815 31606 init.cc:377] Logged in from keytab as kudu/127.28.244.130@KRBTEST.COM (short username kudu)
I20260929 20:28:57.334703 31606 webserver.cc:460] Webserver started at http://127.28.244.130:46803/ using document root <none> and password file <none>
I20260929 20:28:57.335561 31606 fs_manager.cc:359] Metadata directory not provided
I20260929 20:28:57.335711 31606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:28:57.336151 31606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:28:57.339461 31606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-1/data/instance:
uuid: "69e7370ce6e948de9dd613dbe40af8a7"
format_stamp: "Formatted at 2026-09-29 20:28:57 on dist-test-slave-p29l"
server_key: "3f1534401a3a1b4e52a01d7bb48a2bac"
server_key_iv: "e5d2886c96143ca719cfa2d2ec443b35"
server_key_version: "encryptionkey@0"
I20260929 20:28:57.340466 31606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance:
uuid: "69e7370ce6e948de9dd613dbe40af8a7"
format_stamp: "Formatted at 2026-09-29 20:28:57 on dist-test-slave-p29l"
server_key: "3f1534401a3a1b4e52a01d7bb48a2bac"
server_key_iv: "e5d2886c96143ca719cfa2d2ec443b35"
server_key_version: "encryptionkey@0"
I20260929 20:28:57.347098 31606 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260929 20:28:57.351485 31621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:57.353327 31606 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260929 20:28:57.353530 31606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-1/wal
uuid: "69e7370ce6e948de9dd613dbe40af8a7"
format_stamp: "Formatted at 2026-09-29 20:28:57 on dist-test-slave-p29l"
server_key: "3f1534401a3a1b4e52a01d7bb48a2bac"
server_key_iv: "e5d2886c96143ca719cfa2d2ec443b35"
server_key_version: "encryptionkey@0"
I20260929 20:28:57.353765 31606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:28:57.377779 31606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:28:57.382546 31606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:28:57.382884 31606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:28:57.385835 31606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:28:57.385937 31606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:57.386108 31606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:28:57.386198 31606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:28:57 dist-test-slave-p29l krb5kdc[31350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713737, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:28:57.413712 31630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:57.386692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:32805 (local address 127.28.244.190:37129)
0929 20:28:57.387195 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:57.387201 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:57.388705 (+ 1504us) server_negotiation.cc:407] Connection header received
0929 20:28:57.391559 (+ 2854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:57.391565 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:57.391674 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:57.391836 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:57.395208 (+ 3372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:57.396087 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:57.398938 (+ 2851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:57.399604 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:57.399756 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:57.405057 (+ 5301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:57.405065 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:57.405073 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:57.405130 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:57.409317 (+ 4187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:57.410515 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:57.410525 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:57.410531 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:57.410730 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:57.411426 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:57.411432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:57.411436 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:57.411779 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:57.412001 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:57.413034 (+ 1033us) server_negotiation.cc:299] Negotiation successful
0929 20:28:57.413315 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":259,"threads_started":1}
I20260929 20:28:57.414891 31631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:57.387460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37129 (local address 127.28.244.130:32805)
0929 20:28:57.388448 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:57.388530 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:57.391148 (+ 2618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:57.392233 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:57.392251 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:57.393424 (+ 1173us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:28:57.394890 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:57.394945 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:57.396270 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:57.396276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:57.398738 (+ 2462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:57.398757 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:57.399985 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:57.400012 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:57.400432 (+ 420us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:57.401419 (+ 987us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:28:57.401470 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:28:57.404714 (+ 3244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:28:57.409504 (+ 4790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:57.409514 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:57.409538 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:57.410313 (+ 775us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:57.410907 (+ 594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:57.410915 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:57.410920 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:57.411275 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:57.411965 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:28:57.411974 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:28:57.412605 (+ 631us) client_negotiation.cc:769] Sending connection context
0929 20:28:57.412944 (+ 339us) client_negotiation.cc:240] Negotiation successful
0929 20:28:57.413594 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":438,"threads_started":1}
I20260929 20:28:57.440574 31630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:57.422177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:47393 (local address 127.28.244.190:37129)
0929 20:28:57.422424 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:57.422431 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:57.424580 (+ 2149us) server_negotiation.cc:407] Connection header received
0929 20:28:57.424809 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:57.424813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:57.424886 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:57.425039 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:57.426407 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:57.427302 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:57.429098 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:57.429644 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:57.429803 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:57.432897 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:57.432907 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:57.432912 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:57.432961 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:57.436398 (+ 3437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:57.437801 (+ 1403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:57.437809 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:57.437814 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:57.438039 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:57.438666 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:57.438673 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:57.438678 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:57.439208 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:57.439424 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:57.440138 (+ 714us) server_negotiation.cc:299] Negotiation successful
0929 20:28:57.440342 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 20:28:57.440930 31704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:57.422179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37129 (local address 127.28.244.130:47393)
0929 20:28:57.424419 (+ 2240us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:57.424447 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:57.424667 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:57.425057 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:57.425066 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:57.425912 (+ 846us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:28:57.426223 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:57.426233 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:57.427462 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:57.427469 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:57.428923 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:57.428939 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:57.429925 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:57.429935 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:57.430226 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:57.430999 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:28:57.431025 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:28:57.432621 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:28:57.436585 (+ 3964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:57.436594 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:57.436605 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:57.437584 (+ 979us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:57.438206 (+ 622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:57.438215 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:57.438220 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:57.438436 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:57.439373 (+ 937us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:28:57.439382 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:28:57.439731 (+ 349us) client_negotiation.cc:769] Sending connection context
0929 20:28:57.439987 (+ 256us) client_negotiation.cc:240] Negotiation successful
0929 20:28:57.440363 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2018,"thread_start_us":1849,"threads_started":1}
I20260929 20:28:57.444734 31606 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.130:46299
I20260929 20:28:57.445020 31741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.130:46299 every 8 connection(s)
I20260929 20:28:57.446084 31606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
I20260929 20:28:57.448683 31742 heartbeater.cc:347] Connected to a master server at 127.28.244.190:37129
I20260929 20:28:57.448964 31742 heartbeater.cc:464] Registering TS with master...
I20260929 20:28:57.449339 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 31606
I20260929 20:28:57.449519 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance
I20260929 20:28:57.449766 31742 heartbeater.cc:511] Master 127.28.244.190:37129 requested a full tablet report, sending...
I20260929 20:28:57.450006 29650 external_mini_cluster.cc:1428] Setting key 153f1e6a30103164788a37519ea00186
I20260929 20:28:57.451356 31395 ts_manager.cc:194] Registered new tserver with Master: 69e7370ce6e948de9dd613dbe40af8a7 (127.28.244.130:46299)
I20260929 20:28:57.452500 31395 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.130@KRBTEST.COM'} at 127.28.244.130:32805
WARNING: no policy specified for kudu/127.28.244.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab.
Entry for principal kudu/127.28.244.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab.
I20260929 20:28:57.494987 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.131:0
--local_ip_for_outbound_sockets=127.28.244.131
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:37129
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:45291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:28:57.845108 31746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:28:57.845435 31746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:28:57.845541 31746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:28:57.853605 31746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:28:57.853724 31746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:28:57.853811 31746 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 20:28:57.853866 31746 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:28:57.853940 31746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.131
I20260929 20:28:57.865000 31746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.244.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.244.190:37129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.131
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:28:57.866420 31746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:28:57.867780 31746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:28:57.882287 31752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:57.882570 31751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:57.882877 31754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:28:57.883318 31746 server_base.cc:1034] running on GCE node
I20260929 20:28:57.884131 31746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:28:57.885182 31746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:28:57.886497 31746 hybrid_clock.cc:648] HybridClock initialized: now 1790713737886420 us; error 66 us; skew 500 ppm
I20260929 20:28:57.886868 31746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:28:57 dist-test-slave-p29l krb5kdc[31350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713737, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:28:57.890499 31746 init.cc:377] Logged in from keytab as kudu/127.28.244.131@KRBTEST.COM (short username kudu)
I20260929 20:28:57.892400 31746 webserver.cc:460] Webserver started at http://127.28.244.131:45095/ using document root <none> and password file <none>
I20260929 20:28:57.892656 31602 heartbeater.cc:503] Master 127.28.244.190:37129 was elected leader, sending a full tablet report...
I20260929 20:28:57.893267 31746 fs_manager.cc:359] Metadata directory not provided
I20260929 20:28:57.893415 31746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:28:57.893749 31746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:28:57.897341 31746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-2/data/instance:
uuid: "218460624e544e9eb41d3cd524c8e9a7"
format_stamp: "Formatted at 2026-09-29 20:28:57 on dist-test-slave-p29l"
server_key: "ef824cb2a287ad03e8374bb59ff223be"
server_key_iv: "2636b1a6e5b3a6ce971139f87852b431"
server_key_version: "encryptionkey@0"
I20260929 20:28:57.898216 31746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance:
uuid: "218460624e544e9eb41d3cd524c8e9a7"
format_stamp: "Formatted at 2026-09-29 20:28:57 on dist-test-slave-p29l"
server_key: "ef824cb2a287ad03e8374bb59ff223be"
server_key_iv: "2636b1a6e5b3a6ce971139f87852b431"
server_key_version: "encryptionkey@0"
I20260929 20:28:57.904150 31746 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 20:28:57.908110 31761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:57.909696 31746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:28:57.909932 31746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-2/wal
uuid: "218460624e544e9eb41d3cd524c8e9a7"
format_stamp: "Formatted at 2026-09-29 20:28:57 on dist-test-slave-p29l"
server_key: "ef824cb2a287ad03e8374bb59ff223be"
server_key_iv: "2636b1a6e5b3a6ce971139f87852b431"
server_key_version: "encryptionkey@0"
I20260929 20:28:57.910167 31746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:28:57.931224 31746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:28:57.936190 31746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:28:57.936517 31746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:28:57.939519 31746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:28:57.939699 31746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:57.939854 31746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:28:57.939970 31746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 29 20:28:57 dist-test-slave-p29l krb5kdc[31350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713737, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:28:57.968016 31630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:57.940184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:38775 (local address 127.28.244.190:37129)
0929 20:28:57.940409 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:57.940415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:57.942009 (+ 1594us) server_negotiation.cc:407] Connection header received
0929 20:28:57.944704 (+ 2695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:57.944713 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:57.944873 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:57.945020 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:57.947865 (+ 2845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:57.949033 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:57.950942 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:57.951697 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:57.951896 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:57.957899 (+ 6003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:57.957906 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:57.957909 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:57.957985 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:57.962826 (+ 4841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:57.964889 (+ 2063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:57.964897 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:57.964901 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:57.965149 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:57.965836 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:57.965843 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:57.965848 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:57.966226 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:57.966444 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:57.967307 (+ 863us) server_negotiation.cc:299] Negotiation successful
0929 20:28:57.967588 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 20:28:57.969240 31770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:57.940822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37129 (local address 127.28.244.131:38775)
0929 20:28:57.941787 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:57.941838 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:57.944240 (+ 2402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:57.945359 (+ 1119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:57.945375 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:57.946369 (+ 994us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:28:57.947588 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:57.947621 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:57.949256 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:57.949267 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:57.950739 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:57.950755 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:57.952036 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:57.952045 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:57.952349 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:57.953328 (+ 979us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:28:57.953358 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:28:57.957521 (+ 4163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:28:57.963075 (+ 5554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:57.963095 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:57.963130 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:57.964593 (+ 1463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:57.965341 (+ 748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:57.965346 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:57.965348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:57.965609 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:57.966386 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:28:57.966397 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:28:57.966856 (+ 459us) client_negotiation.cc:769] Sending connection context
0929 20:28:57.967190 (+ 334us) client_negotiation.cc:240] Negotiation successful
0929 20:28:57.967864 (+ 674us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":402,"threads_started":1}
I20260929 20:28:57.993041 31630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:57.975699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:52875 (local address 127.28.244.190:37129)
0929 20:28:57.975974 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:57.975980 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:57.977149 (+ 1169us) server_negotiation.cc:407] Connection header received
0929 20:28:57.977390 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:57.977396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:57.977480 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:57.977588 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:57.978662 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:57.979457 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:57.981575 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:57.982224 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:57.982404 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:57.985562 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:57.985571 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:57.985578 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:57.985628 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:57.989154 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:57.990381 (+ 1227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:57.990387 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:57.990391 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:57.990629 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:57.991290 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:57.991298 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:57.991304 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:57.991701 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:57.991915 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:57.992373 (+ 458us) server_negotiation.cc:299] Negotiation successful
0929 20:28:57.992632 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:28:57.993147 31836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:57.975766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37129 (local address 127.28.244.131:52875)
0929 20:28:57.977016 (+ 1250us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:57.977036 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:57.977241 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:57.977627 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:57.977632 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:57.978169 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:28:57.978489 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:57.978499 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:57.979624 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:57.979631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:57.981320 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:57.981346 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:57.982733 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:57.982739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:57.982874 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:57.983825 (+ 951us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:28:57.983859 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:28:57.985186 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:28:57.989459 (+ 4273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:57.989468 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:57.989475 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:57.990197 (+ 722us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:57.990808 (+ 611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:57.990817 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:57.990823 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:57.991101 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:57.991877 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:28:57.991882 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:28:57.992097 (+ 215us) client_negotiation.cc:769] Sending connection context
0929 20:28:57.992247 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 20:28:57.992635 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1092,"thread_start_us":407,"threads_started":1}
I20260929 20:28:58.006484 31746 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.131:43369
I20260929 20:28:58.006553 31880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.131:43369 every 8 connection(s)
I20260929 20:28:58.008162 31746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
I20260929 20:28:58.010705 31881 heartbeater.cc:347] Connected to a master server at 127.28.244.190:37129
I20260929 20:28:58.010995 31881 heartbeater.cc:464] Registering TS with master...
I20260929 20:28:58.011801 31881 heartbeater.cc:511] Master 127.28.244.190:37129 requested a full tablet report, sending...
I20260929 20:28:58.013581 31395 ts_manager.cc:194] Registered new tserver with Master: 218460624e544e9eb41d3cd524c8e9a7 (127.28.244.131:43369)
I20260929 20:28:58.014778 31395 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.131@KRBTEST.COM'} at 127.28.244.131:38775
I20260929 20:28:58.017343 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 31746
I20260929 20:28:58.017578 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance
I20260929 20:28:58.018157 29650 external_mini_cluster.cc:1428] Setting key c5a8669888ad8729c21d619fb5d80994
I20260929 20:28:58.022238 29650 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 20:28:58 dist-test-slave-p29l krb5kdc[31350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 20:28:58 dist-test-slave-p29l krb5kdc[31350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:28:58.074743 31630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:58.058466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41744 (local address 127.28.244.190:37129)
0929 20:28:58.058684 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:58.058692 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:58.059014 (+ 322us) server_negotiation.cc:407] Connection header received
0929 20:28:58.059163 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:58.059168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:58.059236 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:58.059327 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:58.061065 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.061964 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.063351 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.063955 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.064092 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:58.068494 (+ 4402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:58.068501 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:58.068504 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:58.068549 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:58.071488 (+ 2939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:58.072694 (+ 1206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:58.072699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:58.072702 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:58.072884 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:58.073283 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:58.073286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:58.073288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:58.073665 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:58.073884 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:58.074236 (+ 352us) server_negotiation.cc:299] Negotiation successful
0929 20:28:58.074477 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 20:28:58.079789 31395 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:28:58.087181 31395 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:28:58.117686 31897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:58.101863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44320 (local address 127.28.244.131:43369)
0929 20:28:58.103867 (+ 2004us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:58.103875 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:58.104479 (+ 604us) server_negotiation.cc:407] Connection header received
0929 20:28:58.104862 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:58.104872 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:58.105181 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:58.105406 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:28:58.106684 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.109085 (+ 2401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.112933 (+ 3848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.115127 (+ 2194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.115351 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:58.115621 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:58.116820 (+ 1199us) server_negotiation.cc:299] Negotiation successful
0929 20:28:58.117115 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1802,"thread_start_us":1617,"threads_started":1}
I20260929 20:28:58.117897 31895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:58.101337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40472 (local address 127.28.244.130:46299)
0929 20:28:58.102075 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:58.102089 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:58.103028 (+ 939us) server_negotiation.cc:407] Connection header received
0929 20:28:58.103348 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:58.103363 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:58.103761 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:58.104041 (+ 280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:28:58.105680 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.107771 (+ 2091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.112232 (+ 4461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.115347 (+ 3115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.115608 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:58.115861 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:58.117127 (+ 1266us) server_negotiation.cc:299] Negotiation successful
0929 20:28:58.117396 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":325,"threads_started":1}
I20260929 20:28:58.118893 31896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:58.101193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:46299 (local address 127.0.0.1:40472)
0929 20:28:58.102824 (+ 1631us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:58.102861 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:58.103077 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:58.104318 (+ 1241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:58.104324 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:58.104351 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:28:58.105396 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:58.105422 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.108041 (+ 2619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.108051 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:58.111906 (+ 3855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:58.111936 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.116067 (+ 4131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.116077 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:58.116522 (+ 445us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:58.116580 (+ 58us) client_negotiation.cc:769] Sending connection context
0929 20:28:58.116703 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 20:28:58.118181 (+ 1478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1384,"mutex_wait_us":90,"thread_start_us":2841,"threads_started":1}
I20260929 20:28:58.119620 31894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:58.101327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:43369 (local address 127.0.0.1:44320)
0929 20:28:58.104325 (+ 2998us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:58.104345 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:58.104640 (+ 295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:58.105848 (+ 1208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:58.105856 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:58.105910 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:28:58.106387 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:58.106402 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.109345 (+ 2943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.109353 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:58.112629 (+ 3276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:58.112655 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.115929 (+ 3274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.115939 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:58.116397 (+ 458us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:58.116517 (+ 120us) client_negotiation.cc:769] Sending connection context
0929 20:28:58.118509 (+ 1992us) client_negotiation.cc:240] Negotiation successful
0929 20:28:58.118758 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2863,"mutex_wait_us":65,"thread_start_us":1117,"threads_started":1}
I20260929 20:28:58.123888 31811 tablet_service.cc:1467] Processing CreateTablet for tablet f4d7a3603bcd4d0c9211ed7c3efe7c13 (DEFAULT_TABLE table=test-table [id=8cb05ad2dfb74fe2b0ba84ab5ca3675d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:28:58.124054 31672 tablet_service.cc:1467] Processing CreateTablet for tablet f4d7a3603bcd4d0c9211ed7c3efe7c13 (DEFAULT_TABLE table=test-table [id=8cb05ad2dfb74fe2b0ba84ab5ca3675d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:28:58.125586 31811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4d7a3603bcd4d0c9211ed7c3efe7c13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:28:58.125720 31672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4d7a3603bcd4d0c9211ed7c3efe7c13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:28:58.128684 31896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:58.104674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:34227 (local address 127.0.0.1:58434)
0929 20:28:58.119184 (+ 14510us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:58.119204 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:58.119340 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:58.121258 (+ 1918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:58.121263 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:58.121293 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:28:58.121501 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:58.121513 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.123442 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.123448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:58.125788 (+ 2340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:58.125805 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.127776 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.127782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:58.128022 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:58.128063 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 20:28:58.128154 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 20:28:58.128317 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14401,"thread_start_us":19334,"threads_started":1}
I20260929 20:28:58.129887 31899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:58.104937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58434 (local address 127.28.244.129:34227)
0929 20:28:58.120331 (+ 15394us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:58.120340 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:58.120371 (+ 31us) server_negotiation.cc:407] Connection header received
0929 20:28:58.120500 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:58.120508 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:58.120781 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:58.121013 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:28:58.121691 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.123193 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.126027 (+ 2834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.127553 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.127751 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:58.129279 (+ 1528us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:58.129366 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 20:28:58.129578 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15164,"thread_start_us":20460,"threads_started":1}
I20260929 20:28:58.134445 31532 tablet_service.cc:1467] Processing CreateTablet for tablet f4d7a3603bcd4d0c9211ed7c3efe7c13 (DEFAULT_TABLE table=test-table [id=8cb05ad2dfb74fe2b0ba84ab5ca3675d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:28:58.136219 31532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4d7a3603bcd4d0c9211ed7c3efe7c13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:28:58.140288 31900 tablet_bootstrap.cc:492] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 218460624e544e9eb41d3cd524c8e9a7: Bootstrap starting.
I20260929 20:28:58.142551 31901 tablet_bootstrap.cc:492] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 69e7370ce6e948de9dd613dbe40af8a7: Bootstrap starting.
I20260929 20:28:58.144821 31900 tablet_bootstrap.cc:654] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 218460624e544e9eb41d3cd524c8e9a7: Neither blocks nor log segments found. Creating new log.
I20260929 20:28:58.146566 31900 log.cc:824] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 218460624e544e9eb41d3cd524c8e9a7: Log is configured to *not* fsync() on all Append() calls
I20260929 20:28:58.148779 31901 tablet_bootstrap.cc:654] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 69e7370ce6e948de9dd613dbe40af8a7: Neither blocks nor log segments found. Creating new log.
I20260929 20:28:58.150573 31901 log.cc:824] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 69e7370ce6e948de9dd613dbe40af8a7: Log is configured to *not* fsync() on all Append() calls
I20260929 20:28:58.165455 31904 tablet_bootstrap.cc:492] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 1fd552a67209408b9d25ed9b4937d293: Bootstrap starting.
I20260929 20:28:58.167503 31901 tablet_bootstrap.cc:492] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 69e7370ce6e948de9dd613dbe40af8a7: No bootstrap required, opened a new log
I20260929 20:28:58.168051 31901 ts_tablet_manager.cc:1397] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 69e7370ce6e948de9dd613dbe40af8a7: Time spent bootstrapping tablet: real 0.026s user 0.015s sys 0.008s
I20260929 20:28:58.168869 31900 tablet_bootstrap.cc:492] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 218460624e544e9eb41d3cd524c8e9a7: No bootstrap required, opened a new log
I20260929 20:28:58.169299 31900 ts_tablet_manager.cc:1397] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 218460624e544e9eb41d3cd524c8e9a7: Time spent bootstrapping tablet: real 0.029s user 0.010s sys 0.016s
I20260929 20:28:58.169539 31904 tablet_bootstrap.cc:654] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 1fd552a67209408b9d25ed9b4937d293: Neither blocks nor log segments found. Creating new log.
I20260929 20:28:58.171088 31904 log.cc:824] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 1fd552a67209408b9d25ed9b4937d293: Log is configured to *not* fsync() on all Append() calls
I20260929 20:28:58.176141 31904 tablet_bootstrap.cc:492] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 1fd552a67209408b9d25ed9b4937d293: No bootstrap required, opened a new log
I20260929 20:28:58.176497 31904 ts_tablet_manager.cc:1397] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 1fd552a67209408b9d25ed9b4937d293: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260929 20:28:58.179723 31901 raft_consensus.cc:348] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 69e7370ce6e948de9dd613dbe40af8a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } } peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } }
I20260929 20:28:58.180243 31901 raft_consensus.cc:374] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 69e7370ce6e948de9dd613dbe40af8a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:28:58.180356 31901 raft_consensus.cc:729] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 69e7370ce6e948de9dd613dbe40af8a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69e7370ce6e948de9dd613dbe40af8a7, State: Initialized, Role: FOLLOWER
I20260929 20:28:58.180922 31900 raft_consensus.cc:348] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 218460624e544e9eb41d3cd524c8e9a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } } peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } }
I20260929 20:28:58.181072 31901 consensus_queue.cc:260] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 69e7370ce6e948de9dd613dbe40af8a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } } peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } }
I20260929 20:28:58.181416 31900 raft_consensus.cc:374] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 218460624e544e9eb41d3cd524c8e9a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:28:58.181559 31900 raft_consensus.cc:729] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 218460624e544e9eb41d3cd524c8e9a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218460624e544e9eb41d3cd524c8e9a7, State: Initialized, Role: FOLLOWER
I20260929 20:28:58.182287 31742 heartbeater.cc:503] Master 127.28.244.190:37129 was elected leader, sending a full tablet report...
I20260929 20:28:58.182351 31900 consensus_queue.cc:260] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 218460624e544e9eb41d3cd524c8e9a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } } peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } }
I20260929 20:28:58.183079 31901 ts_tablet_manager.cc:1428] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 69e7370ce6e948de9dd613dbe40af8a7: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260929 20:28:58.183622 31881 heartbeater.cc:503] Master 127.28.244.190:37129 was elected leader, sending a full tablet report...
I20260929 20:28:58.184453 31900 ts_tablet_manager.cc:1428] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 218460624e544e9eb41d3cd524c8e9a7: Time spent starting tablet: real 0.015s user 0.012s sys 0.002s
I20260929 20:28:58.190188 31904 raft_consensus.cc:348] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 1fd552a67209408b9d25ed9b4937d293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } } peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } }
I20260929 20:28:58.190703 31904 raft_consensus.cc:374] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 1fd552a67209408b9d25ed9b4937d293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:28:58.190824 31904 raft_consensus.cc:729] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 1fd552a67209408b9d25ed9b4937d293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fd552a67209408b9d25ed9b4937d293, State: Initialized, Role: FOLLOWER
I20260929 20:28:58.191529 31904 consensus_queue.cc:260] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 1fd552a67209408b9d25ed9b4937d293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } } peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } }
I20260929 20:28:58.193552 31904 ts_tablet_manager.cc:1428] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 1fd552a67209408b9d25ed9b4937d293: Time spent starting tablet: real 0.017s user 0.010s sys 0.004s
W20260929 20:28:58.198256 31743 tablet.cc:2369] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 69e7370ce6e948de9dd613dbe40af8a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:28:58.210575 31907 raft_consensus.cc:482] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 218460624e544e9eb41d3cd524c8e9a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:28:58.210906 31907 raft_consensus.cc:504] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 218460624e544e9eb41d3cd524c8e9a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } } peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } }
I20260929 20:28:58.212818 31907 leader_election.cc:288] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 218460624e544e9eb41d3cd524c8e9a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69e7370ce6e948de9dd613dbe40af8a7 (127.28.244.130:46299), 1fd552a67209408b9d25ed9b4937d293 (127.28.244.129:34227)
I20260929 20:28:58.219254 31899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:58.213467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:56381 (local address 127.28.244.129:34227)
0929 20:28:58.213696 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:58.213700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:58.213721 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:28:58.213819 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:58.213826 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:58.213929 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:58.214041 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:28:58.214578 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.215622 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.217432 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.218324 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.218485 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:58.218563 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:58.218860 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 20:28:58.218992 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 20:28:58.219261 31770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:58.213120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:34227 (local address 127.28.244.131:56381)
0929 20:28:58.213356 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:58.213379 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:58.213533 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:58.214112 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:58.214116 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:58.214142 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:28:58.214412 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:58.214432 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.215747 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.215755 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:58.217252 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:58.217274 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.218485 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.218491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:58.218696 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:58.218722 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 20:28:58.218832 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 20:28:58.218967 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 20:28:58.220038 31895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:58.213431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:57635 (local address 127.28.244.130:46299)
0929 20:28:58.213640 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:58.213649 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:58.214394 (+ 745us) server_negotiation.cc:407] Connection header received
0929 20:28:58.214655 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:58.214659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:58.214725 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:58.214797 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:28:58.215416 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.216203 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.217980 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.218952 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.219134 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:58.219210 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:58.219585 (+ 375us) server_negotiation.cc:299] Negotiation successful
0929 20:28:58.219740 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 20:28:58.220443 31557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d7a3603bcd4d0c9211ed7c3efe7c13" candidate_uuid: "218460624e544e9eb41d3cd524c8e9a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fd552a67209408b9d25ed9b4937d293" is_pre_election: true
I20260929 20:28:58.220767 31909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:58.213123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:46299 (local address 127.28.244.131:57635)
0929 20:28:58.214284 (+ 1161us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:58.214309 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:58.214447 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:58.214843 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:58.214847 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:58.214875 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:28:58.215236 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:58.215250 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.216343 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.216351 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:58.217807 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:58.217826 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.219177 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.219183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:58.219382 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:58.219428 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 20:28:58.220232 (+ 804us) client_negotiation.cc:240] Negotiation successful
0929 20:28:58.220382 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":977,"mutex_wait_us":414,"thread_start_us":329,"threads_started":1}
I20260929 20:28:58.221063 31557 raft_consensus.cc:2454] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 1fd552a67209408b9d25ed9b4937d293 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 218460624e544e9eb41d3cd524c8e9a7 in term 0.
I20260929 20:28:58.221748 31764 leader_election.cc:302] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 218460624e544e9eb41d3cd524c8e9a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fd552a67209408b9d25ed9b4937d293, 218460624e544e9eb41d3cd524c8e9a7; no voters:
I20260929 20:28:58.221827 31692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d7a3603bcd4d0c9211ed7c3efe7c13" candidate_uuid: "218460624e544e9eb41d3cd524c8e9a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69e7370ce6e948de9dd613dbe40af8a7" is_pre_election: true
I20260929 20:28:58.222167 31907 raft_consensus.cc:2781] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 218460624e544e9eb41d3cd524c8e9a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:28:58.222327 31907 raft_consensus.cc:482] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 218460624e544e9eb41d3cd524c8e9a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:28:58.222455 31907 raft_consensus.cc:3037] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 218460624e544e9eb41d3cd524c8e9a7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:28:58.222411 31692 raft_consensus.cc:2454] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 69e7370ce6e948de9dd613dbe40af8a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 218460624e544e9eb41d3cd524c8e9a7 in term 0.
I20260929 20:28:58.225275 31907 raft_consensus.cc:504] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 218460624e544e9eb41d3cd524c8e9a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } } peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } }
I20260929 20:28:58.225858 31907 leader_election.cc:288] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 218460624e544e9eb41d3cd524c8e9a7 [CANDIDATE]: Term 1 election: Requested vote from peers 69e7370ce6e948de9dd613dbe40af8a7 (127.28.244.130:46299), 1fd552a67209408b9d25ed9b4937d293 (127.28.244.129:34227)
I20260929 20:28:58.226425 31692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d7a3603bcd4d0c9211ed7c3efe7c13" candidate_uuid: "218460624e544e9eb41d3cd524c8e9a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69e7370ce6e948de9dd613dbe40af8a7"
I20260929 20:28:58.226724 31692 raft_consensus.cc:3037] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 69e7370ce6e948de9dd613dbe40af8a7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:28:58.229420 31692 raft_consensus.cc:2454] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 69e7370ce6e948de9dd613dbe40af8a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 218460624e544e9eb41d3cd524c8e9a7 in term 1.
I20260929 20:28:58.229957 31765 leader_election.cc:302] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 218460624e544e9eb41d3cd524c8e9a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 218460624e544e9eb41d3cd524c8e9a7, 69e7370ce6e948de9dd613dbe40af8a7; no voters:
I20260929 20:28:58.229940 31557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d7a3603bcd4d0c9211ed7c3efe7c13" candidate_uuid: "218460624e544e9eb41d3cd524c8e9a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fd552a67209408b9d25ed9b4937d293"
I20260929 20:28:58.230197 31557 raft_consensus.cc:3037] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 1fd552a67209408b9d25ed9b4937d293 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:28:58.230338 31907 raft_consensus.cc:2781] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 218460624e544e9eb41d3cd524c8e9a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:28:58.230865 31907 raft_consensus.cc:686] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 218460624e544e9eb41d3cd524c8e9a7 [term 1 LEADER]: Becoming Leader. State: Replica: 218460624e544e9eb41d3cd524c8e9a7, State: Running, Role: LEADER
I20260929 20:28:58.231407 31907 consensus_queue.cc:237] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 218460624e544e9eb41d3cd524c8e9a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } } peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } }
I20260929 20:28:58.233132 31557 raft_consensus.cc:2454] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 1fd552a67209408b9d25ed9b4937d293 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 218460624e544e9eb41d3cd524c8e9a7 in term 1.
I20260929 20:28:58.236498 31394 catalog_manager.cc:5495] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 218460624e544e9eb41d3cd524c8e9a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 218460624e544e9eb41d3cd524c8e9a7 (127.28.244.131). New cstate: current_term: 1 leader_uuid: "218460624e544e9eb41d3cd524c8e9a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } health_report { overall_health: UNKNOWN } } }
W20260929 20:28:58.260078 31882 tablet.cc:2369] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 218460624e544e9eb41d3cd524c8e9a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 20:28:58 dist-test-slave-p29l krb5kdc[31350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713736, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:28:58.308715 31917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:58.294717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37129 (local address 127.0.0.1:41752)
0929 20:28:58.295365 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:58.295382 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:58.295573 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:58.295964 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:58.295969 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:58.296708 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:28:58.296988 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:58.296999 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.298020 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.298026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:58.299051 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:58.299065 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.299790 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.299796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:58.300009 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:58.300659 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:28:58.300677 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:28:58.303033 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:28:58.306154 (+ 3121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:58.306167 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:58.306173 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:58.306638 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:58.307057 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:58.307061 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:58.307063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:58.307199 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:58.307710 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:28:58.307717 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:28:58.307990 (+ 273us) client_negotiation.cc:769] Sending connection context
0929 20:28:58.308137 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 20:28:58.308383 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":392,"threads_started":1}
I20260929 20:28:58.308849 31630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:58.294707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41752 (local address 127.28.244.190:37129)
0929 20:28:58.294945 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:58.294950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:58.295485 (+ 535us) server_negotiation.cc:407] Connection header received
0929 20:28:58.295703 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:58.295712 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:58.295778 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:58.295926 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:58.297135 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.297876 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.299204 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.299605 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.299745 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:58.303243 (+ 3498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:58.303248 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:58.303251 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:58.303281 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:58.306006 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:58.306812 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:58.306817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:58.306819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:58.306952 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:58.307295 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:58.307298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:58.307300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:58.307592 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:58.307721 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:58.308239 (+ 518us) server_negotiation.cc:299] Negotiation successful
0929 20:28:58.308519 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":95,"server-negotiator.queue_time_us":80}
I20260929 20:28:58.313977 31394 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:41752:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 20:28:58.315405 31394 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:28:58.322657 31672 tablet_service.cc:1467] Processing CreateTablet for tablet 06977840bfb84ea282ddee8d77c4fae2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5a7d1141607f48e98b2e5346002004ac]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:28:58.322657 31532 tablet_service.cc:1467] Processing CreateTablet for tablet 06977840bfb84ea282ddee8d77c4fae2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5a7d1141607f48e98b2e5346002004ac]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:28:58.322911 31811 tablet_service.cc:1467] Processing CreateTablet for tablet 06977840bfb84ea282ddee8d77c4fae2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5a7d1141607f48e98b2e5346002004ac]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:28:58.323707 31532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06977840bfb84ea282ddee8d77c4fae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:28:58.323707 31672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06977840bfb84ea282ddee8d77c4fae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:28:58.323908 31811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06977840bfb84ea282ddee8d77c4fae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:28:58.329960 31900 tablet_bootstrap.cc:492] T 06977840bfb84ea282ddee8d77c4fae2 P 218460624e544e9eb41d3cd524c8e9a7: Bootstrap starting.
I20260929 20:28:58.331157 31904 tablet_bootstrap.cc:492] T 06977840bfb84ea282ddee8d77c4fae2 P 1fd552a67209408b9d25ed9b4937d293: Bootstrap starting.
I20260929 20:28:58.331553 31900 tablet_bootstrap.cc:654] T 06977840bfb84ea282ddee8d77c4fae2 P 218460624e544e9eb41d3cd524c8e9a7: Neither blocks nor log segments found. Creating new log.
I20260929 20:28:58.332984 31901 tablet_bootstrap.cc:492] T 06977840bfb84ea282ddee8d77c4fae2 P 69e7370ce6e948de9dd613dbe40af8a7: Bootstrap starting.
I20260929 20:28:58.333022 31904 tablet_bootstrap.cc:654] T 06977840bfb84ea282ddee8d77c4fae2 P 1fd552a67209408b9d25ed9b4937d293: Neither blocks nor log segments found. Creating new log.
I20260929 20:28:58.333818 31900 tablet_bootstrap.cc:492] T 06977840bfb84ea282ddee8d77c4fae2 P 218460624e544e9eb41d3cd524c8e9a7: No bootstrap required, opened a new log
I20260929 20:28:58.334002 31900 ts_tablet_manager.cc:1397] T 06977840bfb84ea282ddee8d77c4fae2 P 218460624e544e9eb41d3cd524c8e9a7: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 20:28:58.334672 31900 raft_consensus.cc:348] T 06977840bfb84ea282ddee8d77c4fae2 P 218460624e544e9eb41d3cd524c8e9a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } }
I20260929 20:28:58.334913 31900 raft_consensus.cc:374] T 06977840bfb84ea282ddee8d77c4fae2 P 218460624e544e9eb41d3cd524c8e9a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:28:58.335003 31900 raft_consensus.cc:729] T 06977840bfb84ea282ddee8d77c4fae2 P 218460624e544e9eb41d3cd524c8e9a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218460624e544e9eb41d3cd524c8e9a7, State: Initialized, Role: FOLLOWER
I20260929 20:28:58.335130 31901 tablet_bootstrap.cc:654] T 06977840bfb84ea282ddee8d77c4fae2 P 69e7370ce6e948de9dd613dbe40af8a7: Neither blocks nor log segments found. Creating new log.
I20260929 20:28:58.335268 31900 consensus_queue.cc:260] T 06977840bfb84ea282ddee8d77c4fae2 P 218460624e544e9eb41d3cd524c8e9a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } }
I20260929 20:28:58.335743 31907 tablet_replica.cc:439] T 06977840bfb84ea282ddee8d77c4fae2 P 218460624e544e9eb41d3cd524c8e9a7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } } }
I20260929 20:28:58.335992 31900 ts_tablet_manager.cc:1428] T 06977840bfb84ea282ddee8d77c4fae2 P 218460624e544e9eb41d3cd524c8e9a7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 20:28:58.336084 31907 tablet_replica.cc:442] T 06977840bfb84ea282ddee8d77c4fae2 P 218460624e544e9eb41d3cd524c8e9a7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:28:58.341277 31904 tablet_bootstrap.cc:492] T 06977840bfb84ea282ddee8d77c4fae2 P 1fd552a67209408b9d25ed9b4937d293: No bootstrap required, opened a new log
I20260929 20:28:58.341472 31904 ts_tablet_manager.cc:1397] T 06977840bfb84ea282ddee8d77c4fae2 P 1fd552a67209408b9d25ed9b4937d293: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20260929 20:28:58.341463 31901 tablet_bootstrap.cc:492] T 06977840bfb84ea282ddee8d77c4fae2 P 69e7370ce6e948de9dd613dbe40af8a7: No bootstrap required, opened a new log
I20260929 20:28:58.341647 31901 ts_tablet_manager.cc:1397] T 06977840bfb84ea282ddee8d77c4fae2 P 69e7370ce6e948de9dd613dbe40af8a7: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260929 20:28:58.342314 31904 raft_consensus.cc:348] T 06977840bfb84ea282ddee8d77c4fae2 P 1fd552a67209408b9d25ed9b4937d293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } }
I20260929 20:28:58.342399 31901 raft_consensus.cc:348] T 06977840bfb84ea282ddee8d77c4fae2 P 69e7370ce6e948de9dd613dbe40af8a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } }
I20260929 20:28:58.342648 31904 raft_consensus.cc:374] T 06977840bfb84ea282ddee8d77c4fae2 P 1fd552a67209408b9d25ed9b4937d293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:28:58.342679 31901 raft_consensus.cc:374] T 06977840bfb84ea282ddee8d77c4fae2 P 69e7370ce6e948de9dd613dbe40af8a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:28:58.342759 31904 raft_consensus.cc:729] T 06977840bfb84ea282ddee8d77c4fae2 P 1fd552a67209408b9d25ed9b4937d293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fd552a67209408b9d25ed9b4937d293, State: Initialized, Role: FOLLOWER
I20260929 20:28:58.342774 31901 raft_consensus.cc:729] T 06977840bfb84ea282ddee8d77c4fae2 P 69e7370ce6e948de9dd613dbe40af8a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69e7370ce6e948de9dd613dbe40af8a7, State: Initialized, Role: FOLLOWER
I20260929 20:28:58.343006 31901 consensus_queue.cc:260] T 06977840bfb84ea282ddee8d77c4fae2 P 69e7370ce6e948de9dd613dbe40af8a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } }
I20260929 20:28:58.343073 31904 consensus_queue.cc:260] T 06977840bfb84ea282ddee8d77c4fae2 P 1fd552a67209408b9d25ed9b4937d293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } }
I20260929 20:28:58.343631 31901 ts_tablet_manager.cc:1428] T 06977840bfb84ea282ddee8d77c4fae2 P 69e7370ce6e948de9dd613dbe40af8a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 20:28:58.343739 31904 ts_tablet_manager.cc:1428] T 06977840bfb84ea282ddee8d77c4fae2 P 1fd552a67209408b9d25ed9b4937d293: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260929 20:28:58.343461 31906 tablet_replica.cc:439] T 06977840bfb84ea282ddee8d77c4fae2 P 69e7370ce6e948de9dd613dbe40af8a7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } } }
I20260929 20:28:58.343528 31908 tablet_replica.cc:439] T 06977840bfb84ea282ddee8d77c4fae2 P 1fd552a67209408b9d25ed9b4937d293: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } } }
I20260929 20:28:58.343926 31906 tablet_replica.cc:442] T 06977840bfb84ea282ddee8d77c4fae2 P 69e7370ce6e948de9dd613dbe40af8a7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:28:58.343981 31908 tablet_replica.cc:442] T 06977840bfb84ea282ddee8d77c4fae2 P 1fd552a67209408b9d25ed9b4937d293: This TxnStatusTablet replica's current role is: FOLLOWER
W20260929 20:28:58.383159 31603 tablet.cc:2369] T 06977840bfb84ea282ddee8d77c4fae2 P 1fd552a67209408b9d25ed9b4937d293: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:28:58.423931 31907 raft_consensus.cc:482] T 06977840bfb84ea282ddee8d77c4fae2 P 218460624e544e9eb41d3cd524c8e9a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:28:58.424193 31907 raft_consensus.cc:504] T 06977840bfb84ea282ddee8d77c4fae2 P 218460624e544e9eb41d3cd524c8e9a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } }
I20260929 20:28:58.424875 31907 leader_election.cc:288] T 06977840bfb84ea282ddee8d77c4fae2 P 218460624e544e9eb41d3cd524c8e9a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fd552a67209408b9d25ed9b4937d293 (127.28.244.129:34227), 69e7370ce6e948de9dd613dbe40af8a7 (127.28.244.130:46299)
I20260929 20:28:58.425436 31557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06977840bfb84ea282ddee8d77c4fae2" candidate_uuid: "218460624e544e9eb41d3cd524c8e9a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fd552a67209408b9d25ed9b4937d293" is_pre_election: true
I20260929 20:28:58.425462 31692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06977840bfb84ea282ddee8d77c4fae2" candidate_uuid: "218460624e544e9eb41d3cd524c8e9a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69e7370ce6e948de9dd613dbe40af8a7" is_pre_election: true
I20260929 20:28:58.425694 31557 raft_consensus.cc:2454] T 06977840bfb84ea282ddee8d77c4fae2 P 1fd552a67209408b9d25ed9b4937d293 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 218460624e544e9eb41d3cd524c8e9a7 in term 0.
I20260929 20:28:58.425725 31692 raft_consensus.cc:2454] T 06977840bfb84ea282ddee8d77c4fae2 P 69e7370ce6e948de9dd613dbe40af8a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 218460624e544e9eb41d3cd524c8e9a7 in term 0.
I20260929 20:28:58.426110 31764 leader_election.cc:302] T 06977840bfb84ea282ddee8d77c4fae2 P 218460624e544e9eb41d3cd524c8e9a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fd552a67209408b9d25ed9b4937d293, 218460624e544e9eb41d3cd524c8e9a7; no voters:
I20260929 20:28:58.426393 31907 raft_consensus.cc:2781] T 06977840bfb84ea282ddee8d77c4fae2 P 218460624e544e9eb41d3cd524c8e9a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:28:58.426512 31907 raft_consensus.cc:482] T 06977840bfb84ea282ddee8d77c4fae2 P 218460624e544e9eb41d3cd524c8e9a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:28:58.426642 31907 raft_consensus.cc:3037] T 06977840bfb84ea282ddee8d77c4fae2 P 218460624e544e9eb41d3cd524c8e9a7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:28:58.428769 31907 raft_consensus.cc:504] T 06977840bfb84ea282ddee8d77c4fae2 P 218460624e544e9eb41d3cd524c8e9a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } }
I20260929 20:28:58.429402 31907 leader_election.cc:288] T 06977840bfb84ea282ddee8d77c4fae2 P 218460624e544e9eb41d3cd524c8e9a7 [CANDIDATE]: Term 1 election: Requested vote from peers 1fd552a67209408b9d25ed9b4937d293 (127.28.244.129:34227), 69e7370ce6e948de9dd613dbe40af8a7 (127.28.244.130:46299)
I20260929 20:28:58.429924 31692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06977840bfb84ea282ddee8d77c4fae2" candidate_uuid: "218460624e544e9eb41d3cd524c8e9a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69e7370ce6e948de9dd613dbe40af8a7"
I20260929 20:28:58.429924 31557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06977840bfb84ea282ddee8d77c4fae2" candidate_uuid: "218460624e544e9eb41d3cd524c8e9a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fd552a67209408b9d25ed9b4937d293"
I20260929 20:28:58.430195 31692 raft_consensus.cc:3037] T 06977840bfb84ea282ddee8d77c4fae2 P 69e7370ce6e948de9dd613dbe40af8a7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:28:58.430195 31557 raft_consensus.cc:3037] T 06977840bfb84ea282ddee8d77c4fae2 P 1fd552a67209408b9d25ed9b4937d293 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:28:58.433260 31692 raft_consensus.cc:2454] T 06977840bfb84ea282ddee8d77c4fae2 P 69e7370ce6e948de9dd613dbe40af8a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 218460624e544e9eb41d3cd524c8e9a7 in term 1.
I20260929 20:28:58.433260 31557 raft_consensus.cc:2454] T 06977840bfb84ea282ddee8d77c4fae2 P 1fd552a67209408b9d25ed9b4937d293 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 218460624e544e9eb41d3cd524c8e9a7 in term 1.
I20260929 20:28:58.433707 31765 leader_election.cc:302] T 06977840bfb84ea282ddee8d77c4fae2 P 218460624e544e9eb41d3cd524c8e9a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 218460624e544e9eb41d3cd524c8e9a7, 69e7370ce6e948de9dd613dbe40af8a7; no voters:
I20260929 20:28:58.434020 31907 raft_consensus.cc:2781] T 06977840bfb84ea282ddee8d77c4fae2 P 218460624e544e9eb41d3cd524c8e9a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:28:58.434193 31907 raft_consensus.cc:686] T 06977840bfb84ea282ddee8d77c4fae2 P 218460624e544e9eb41d3cd524c8e9a7 [term 1 LEADER]: Becoming Leader. State: Replica: 218460624e544e9eb41d3cd524c8e9a7, State: Running, Role: LEADER
I20260929 20:28:58.434477 31907 consensus_queue.cc:237] T 06977840bfb84ea282ddee8d77c4fae2 P 218460624e544e9eb41d3cd524c8e9a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } }
I20260929 20:28:58.436187 31910 tablet_replica.cc:439] T 06977840bfb84ea282ddee8d77c4fae2 P 218460624e544e9eb41d3cd524c8e9a7: TxnStatusTablet state changed. Reason: New leader 218460624e544e9eb41d3cd524c8e9a7. Latest consensus state: current_term: 1 leader_uuid: "218460624e544e9eb41d3cd524c8e9a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } } }
I20260929 20:28:58.436589 31910 tablet_replica.cc:442] T 06977840bfb84ea282ddee8d77c4fae2 P 218460624e544e9eb41d3cd524c8e9a7: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:28:58.437378 31922 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 20:28:58.437702 31922 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 20:28:58.438030 31395 catalog_manager.cc:5495] T 06977840bfb84ea282ddee8d77c4fae2 P 218460624e544e9eb41d3cd524c8e9a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 218460624e544e9eb41d3cd524c8e9a7 (127.28.244.131). New cstate: current_term: 1 leader_uuid: "218460624e544e9eb41d3cd524c8e9a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } health_report { overall_health: HEALTHY } } }
I20260929 20:28:58.466521 31917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:58.458208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:43369 (local address 127.0.0.1:44326)
0929 20:28:58.458481 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:58.458501 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:58.458706 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:58.459343 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:58.459348 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:58.459361 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:28:58.459554 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:58.459563 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.460862 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.460867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:58.461997 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:58.462016 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.463139 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.463149 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:58.463614 (+ 465us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:58.463774 (+ 160us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:28:58.465670 (+ 1896us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:28:58.465735 (+ 65us) client_negotiation.cc:769] Sending connection context
0929 20:28:58.465896 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 20:28:58.466129 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 20:28:58.466507 31897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:58.458229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44326 (local address 127.28.244.131:43369)
0929 20:28:58.458438 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:58.458445 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:58.458914 (+ 469us) server_negotiation.cc:407] Connection header received
0929 20:28:58.459050 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:58.459057 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:58.459156 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:58.459318 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:28:58.459712 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.460697 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.462169 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.462688 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.463074 (+ 386us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:58.464355 (+ 1281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:28:58.465282 (+ 927us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:28:58.465777 (+ 495us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:58.465942 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 20:28:58.466070 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 20:28:58.476871 31557 raft_consensus.cc:1261] T 06977840bfb84ea282ddee8d77c4fae2 P 1fd552a67209408b9d25ed9b4937d293 [term 1 FOLLOWER]: Refusing update from remote peer 218460624e544e9eb41d3cd524c8e9a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:28:58.477097 31692 raft_consensus.cc:1261] T 06977840bfb84ea282ddee8d77c4fae2 P 69e7370ce6e948de9dd613dbe40af8a7 [term 1 FOLLOWER]: Refusing update from remote peer 218460624e544e9eb41d3cd524c8e9a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:28:58.478099 31910 consensus_queue.cc:1035] T 06977840bfb84ea282ddee8d77c4fae2 P 218460624e544e9eb41d3cd524c8e9a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:28:58.478461 31907 consensus_queue.cc:1035] T 06977840bfb84ea282ddee8d77c4fae2 P 218460624e544e9eb41d3cd524c8e9a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:28:58.492383 31908 tablet_replica.cc:439] T 06977840bfb84ea282ddee8d77c4fae2 P 1fd552a67209408b9d25ed9b4937d293: TxnStatusTablet state changed. Reason: New leader 218460624e544e9eb41d3cd524c8e9a7. Latest consensus state: current_term: 1 leader_uuid: "218460624e544e9eb41d3cd524c8e9a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } } }
I20260929 20:28:58.492729 31908 tablet_replica.cc:442] T 06977840bfb84ea282ddee8d77c4fae2 P 1fd552a67209408b9d25ed9b4937d293: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:28:58.495086 31925 mvcc.cc:205] Tried to move back new op lower bound from 7334763472790011904 to 7334763472630042624. Current Snapshot: MvccSnapshot[applied={T|T < 7334763472790011904}]
I20260929 20:28:58.495447 31906 tablet_replica.cc:439] T 06977840bfb84ea282ddee8d77c4fae2 P 69e7370ce6e948de9dd613dbe40af8a7: TxnStatusTablet state changed. Reason: New leader 218460624e544e9eb41d3cd524c8e9a7. Latest consensus state: current_term: 1 leader_uuid: "218460624e544e9eb41d3cd524c8e9a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } } }
I20260929 20:28:58.495774 31906 tablet_replica.cc:442] T 06977840bfb84ea282ddee8d77c4fae2 P 69e7370ce6e948de9dd613dbe40af8a7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:28:58.498526 31907 tablet_replica.cc:439] T 06977840bfb84ea282ddee8d77c4fae2 P 218460624e544e9eb41d3cd524c8e9a7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "218460624e544e9eb41d3cd524c8e9a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } } }
I20260929 20:28:58.498847 31907 tablet_replica.cc:442] T 06977840bfb84ea282ddee8d77c4fae2 P 218460624e544e9eb41d3cd524c8e9a7: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:28:58.499107 31907 tablet_replica.cc:439] T 06977840bfb84ea282ddee8d77c4fae2 P 218460624e544e9eb41d3cd524c8e9a7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "218460624e544e9eb41d3cd524c8e9a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } } }
I20260929 20:28:58.499383 31907 tablet_replica.cc:442] T 06977840bfb84ea282ddee8d77c4fae2 P 218460624e544e9eb41d3cd524c8e9a7: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:28:58.507467 31908 tablet_replica.cc:439] T 06977840bfb84ea282ddee8d77c4fae2 P 1fd552a67209408b9d25ed9b4937d293: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "218460624e544e9eb41d3cd524c8e9a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } } }
I20260929 20:28:58.507884 31908 tablet_replica.cc:442] T 06977840bfb84ea282ddee8d77c4fae2 P 1fd552a67209408b9d25ed9b4937d293: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:28:58.514472 31906 tablet_replica.cc:439] T 06977840bfb84ea282ddee8d77c4fae2 P 69e7370ce6e948de9dd613dbe40af8a7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "218460624e544e9eb41d3cd524c8e9a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } } }
I20260929 20:28:58.514756 31906 tablet_replica.cc:442] T 06977840bfb84ea282ddee8d77c4fae2 P 69e7370ce6e948de9dd613dbe40af8a7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:28:58.520120 31897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:58.514319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44338 (local address 127.28.244.131:43369)
0929 20:28:58.514533 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:58.514539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:58.514558 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:28:58.514623 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:58.514629 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:58.514718 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:58.514819 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:28:58.515446 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.516609 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.518224 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.518673 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.518838 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:58.519291 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:28:58.519422 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:28:58.519531 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:58.519696 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 20:28:58.519781 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 20:28:58.537722 31897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:58.531711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:58041 (local address 127.28.244.131:43369)
0929 20:28:58.531907 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:58.531912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:58.532588 (+ 676us) server_negotiation.cc:407] Connection header received
0929 20:28:58.532836 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:58.532841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:58.532906 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:58.532996 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:28:58.533545 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.534317 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.535782 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.536273 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.536432 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:58.536896 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:28:58.537077 (+ 181us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:28:58.537177 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:58.537306 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 20:28:58.537377 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 20:28:58.537845 31936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:58.531768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:43369 (local address 127.28.244.131:58041)
0929 20:28:58.532472 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:58.532493 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:58.532670 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:58.533013 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:58.533021 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:58.533031 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:28:58.533381 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:58.533390 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.534458 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.534465 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:58.535632 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:58.535645 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.536478 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.536486 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:58.536616 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:58.536679 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:28:58.537175 (+ 496us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:28:58.537186 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 20:28:58.537288 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 20:28:58.537427 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":297,"threads_started":1}
I20260929 20:28:58.549356 31692 raft_consensus.cc:1261] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 69e7370ce6e948de9dd613dbe40af8a7 [term 1 FOLLOWER]: Refusing update from remote peer 218460624e544e9eb41d3cd524c8e9a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:28:58.549557 31556 raft_consensus.cc:1261] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 1fd552a67209408b9d25ed9b4937d293 [term 1 FOLLOWER]: Refusing update from remote peer 218460624e544e9eb41d3cd524c8e9a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:28:58.550173 31907 consensus_queue.cc:1035] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 218460624e544e9eb41d3cd524c8e9a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:28:58.550649 31907 consensus_queue.cc:1035] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 218460624e544e9eb41d3cd524c8e9a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:28:58.562120 31926 mvcc.cc:205] Tried to move back new op lower bound from 7334763473089372160 to 7334763471800475648. Current Snapshot: MvccSnapshot[applied={T|T < 7334763473089372160}]
I20260929 20:28:58.625672 31395 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:28:58.625921 31395 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:28:58.629884 31395 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5884f7b08d4841ba82729ea2581cdd91: Sending DeleteTablet for 3 replicas of tablet f4d7a3603bcd4d0c9211ed7c3efe7c13
I20260929 20:28:58.631196 31811 tablet_service.cc:1514] Processing DeleteTablet for tablet f4d7a3603bcd4d0c9211ed7c3efe7c13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:28:58 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:44320
I20260929 20:28:58.631160 31672 tablet_service.cc:1514] Processing DeleteTablet for tablet f4d7a3603bcd4d0c9211ed7c3efe7c13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:28:58 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:40472
I20260929 20:28:58.631995 31532 tablet_service.cc:1514] Processing DeleteTablet for tablet f4d7a3603bcd4d0c9211ed7c3efe7c13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:28:58 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:58434
I20260929 20:28:58.633136 31395 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:28:58.634091 31395 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:28:58.638685 31944 tablet_replica.cc:330] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 69e7370ce6e948de9dd613dbe40af8a7: stopping tablet replica
I20260929 20:28:58.638942 31943 tablet_replica.cc:330] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 218460624e544e9eb41d3cd524c8e9a7: stopping tablet replica
I20260929 20:28:58.639187 31944 raft_consensus.cc:2229] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 69e7370ce6e948de9dd613dbe40af8a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:28:58.639364 31943 raft_consensus.cc:2229] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 218460624e544e9eb41d3cd524c8e9a7 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:28:58.639606 31944 raft_consensus.cc:2258] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 69e7370ce6e948de9dd613dbe40af8a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:28:58.640304 31943 raft_consensus.cc:2258] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 218460624e544e9eb41d3cd524c8e9a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:28:58.642583 31943 ts_tablet_manager.cc:1905] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 218460624e544e9eb41d3cd524c8e9a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:28:58.642745 31944 ts_tablet_manager.cc:1905] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 69e7370ce6e948de9dd613dbe40af8a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:28:58.646849 31672 tablet_service.cc:1467] Processing CreateTablet for tablet bb5b9a4eb9b0470ba655d5e55f8e9e7a (DEFAULT_TABLE table=test-table [id=be919f9c0ab54b8f9e12b2cb98dc6624]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:28:58.647339 31672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb5b9a4eb9b0470ba655d5e55f8e9e7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:28:58.647204 31531 tablet_service.cc:1467] Processing CreateTablet for tablet bb5b9a4eb9b0470ba655d5e55f8e9e7a (DEFAULT_TABLE table=test-table [id=be919f9c0ab54b8f9e12b2cb98dc6624]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:28:58.647825 31531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb5b9a4eb9b0470ba655d5e55f8e9e7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:28:58.649335 31810 tablet_service.cc:1467] Processing CreateTablet for tablet bb5b9a4eb9b0470ba655d5e55f8e9e7a (DEFAULT_TABLE table=test-table [id=be919f9c0ab54b8f9e12b2cb98dc6624]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:28:58.649820 31810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb5b9a4eb9b0470ba655d5e55f8e9e7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:28:58.650781 31944 ts_tablet_manager.cc:1918] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 69e7370ce6e948de9dd613dbe40af8a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:28:58.650965 31944 log.cc:1196] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 69e7370ce6e948de9dd613dbe40af8a7: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-1/wal/wals/f4d7a3603bcd4d0c9211ed7c3efe7c13
I20260929 20:28:58.651537 31944 ts_tablet_manager.cc:1939] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 69e7370ce6e948de9dd613dbe40af8a7: Deleting consensus metadata
I20260929 20:28:58.653628 31382 catalog_manager.cc:4841] TS 69e7370ce6e948de9dd613dbe40af8a7 (127.28.244.130:46299): tablet f4d7a3603bcd4d0c9211ed7c3efe7c13 (table test-table [id=8cb05ad2dfb74fe2b0ba84ab5ca3675d]) successfully deleted
I20260929 20:28:58.654047 31901 tablet_bootstrap.cc:492] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 69e7370ce6e948de9dd613dbe40af8a7: Bootstrap starting.
I20260929 20:28:58.654495 31904 tablet_bootstrap.cc:492] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 1fd552a67209408b9d25ed9b4937d293: Bootstrap starting.
I20260929 20:28:58.655737 31901 tablet_bootstrap.cc:654] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 69e7370ce6e948de9dd613dbe40af8a7: Neither blocks nor log segments found. Creating new log.
I20260929 20:28:58.656221 31904 tablet_bootstrap.cc:654] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 1fd552a67209408b9d25ed9b4937d293: Neither blocks nor log segments found. Creating new log.
I20260929 20:28:58.658263 31901 tablet_bootstrap.cc:492] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 69e7370ce6e948de9dd613dbe40af8a7: No bootstrap required, opened a new log
I20260929 20:28:58.658439 31901 ts_tablet_manager.cc:1397] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 69e7370ce6e948de9dd613dbe40af8a7: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260929 20:28:58.659248 31901 raft_consensus.cc:348] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 69e7370ce6e948de9dd613dbe40af8a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } }
I20260929 20:28:58.659548 31901 raft_consensus.cc:374] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 69e7370ce6e948de9dd613dbe40af8a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:28:58.659627 31901 raft_consensus.cc:729] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 69e7370ce6e948de9dd613dbe40af8a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69e7370ce6e948de9dd613dbe40af8a7, State: Initialized, Role: FOLLOWER
I20260929 20:28:58.660091 31901 consensus_queue.cc:260] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 69e7370ce6e948de9dd613dbe40af8a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } }
I20260929 20:28:58.660537 31943 ts_tablet_manager.cc:1918] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 218460624e544e9eb41d3cd524c8e9a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:28:58.660732 31943 log.cc:1196] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 218460624e544e9eb41d3cd524c8e9a7: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-2/wal/wals/f4d7a3603bcd4d0c9211ed7c3efe7c13
I20260929 20:28:58.660737 31901 ts_tablet_manager.cc:1428] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 69e7370ce6e948de9dd613dbe40af8a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 20:28:58.661311 31943 ts_tablet_manager.cc:1939] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 218460624e544e9eb41d3cd524c8e9a7: Deleting consensus metadata
I20260929 20:28:58.663906 31945 tablet_replica.cc:330] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 1fd552a67209408b9d25ed9b4937d293: stopping tablet replica
I20260929 20:28:58.664481 31380 catalog_manager.cc:4841] TS 218460624e544e9eb41d3cd524c8e9a7 (127.28.244.131:43369): tablet f4d7a3603bcd4d0c9211ed7c3efe7c13 (table test-table [id=8cb05ad2dfb74fe2b0ba84ab5ca3675d]) successfully deleted
I20260929 20:28:58.665050 31904 tablet_bootstrap.cc:492] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 1fd552a67209408b9d25ed9b4937d293: No bootstrap required, opened a new log
I20260929 20:28:58.665285 31904 ts_tablet_manager.cc:1397] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 1fd552a67209408b9d25ed9b4937d293: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260929 20:28:58.666251 31945 raft_consensus.cc:2229] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 1fd552a67209408b9d25ed9b4937d293 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:28:58.666168 31904 raft_consensus.cc:348] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 1fd552a67209408b9d25ed9b4937d293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } }
I20260929 20:28:58.666533 31904 raft_consensus.cc:374] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 1fd552a67209408b9d25ed9b4937d293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:28:58.666640 31904 raft_consensus.cc:729] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 1fd552a67209408b9d25ed9b4937d293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fd552a67209408b9d25ed9b4937d293, State: Initialized, Role: FOLLOWER
I20260929 20:28:58.666836 31945 raft_consensus.cc:2258] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 1fd552a67209408b9d25ed9b4937d293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:28:58.667153 31900 tablet_bootstrap.cc:492] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 218460624e544e9eb41d3cd524c8e9a7: Bootstrap starting.
I20260929 20:28:58.668110 31904 consensus_queue.cc:260] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 1fd552a67209408b9d25ed9b4937d293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } }
I20260929 20:28:58.669091 31900 tablet_bootstrap.cc:654] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 218460624e544e9eb41d3cd524c8e9a7: Neither blocks nor log segments found. Creating new log.
I20260929 20:28:58.669883 31904 ts_tablet_manager.cc:1428] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 1fd552a67209408b9d25ed9b4937d293: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260929 20:28:58.671573 31900 tablet_bootstrap.cc:492] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 218460624e544e9eb41d3cd524c8e9a7: No bootstrap required, opened a new log
I20260929 20:28:58.671772 31900 ts_tablet_manager.cc:1397] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 218460624e544e9eb41d3cd524c8e9a7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 20:28:58.672664 31900 raft_consensus.cc:348] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 218460624e544e9eb41d3cd524c8e9a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } }
I20260929 20:28:58.672935 31900 raft_consensus.cc:374] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 218460624e544e9eb41d3cd524c8e9a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:28:58.673020 31900 raft_consensus.cc:729] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 218460624e544e9eb41d3cd524c8e9a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218460624e544e9eb41d3cd524c8e9a7, State: Initialized, Role: FOLLOWER
I20260929 20:28:58.673305 31900 consensus_queue.cc:260] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 218460624e544e9eb41d3cd524c8e9a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } }
I20260929 20:28:58.674095 31900 ts_tablet_manager.cc:1428] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 218460624e544e9eb41d3cd524c8e9a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 20:28:58.674912 31945 ts_tablet_manager.cc:1905] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 1fd552a67209408b9d25ed9b4937d293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:28:58.681253 31945 ts_tablet_manager.cc:1918] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 1fd552a67209408b9d25ed9b4937d293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:28:58.681475 31945 log.cc:1196] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 1fd552a67209408b9d25ed9b4937d293: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-0/wal/wals/f4d7a3603bcd4d0c9211ed7c3efe7c13
I20260929 20:28:58.681990 31945 ts_tablet_manager.cc:1939] T f4d7a3603bcd4d0c9211ed7c3efe7c13 P 1fd552a67209408b9d25ed9b4937d293: Deleting consensus metadata
I20260929 20:28:58.683701 31381 catalog_manager.cc:4841] TS 1fd552a67209408b9d25ed9b4937d293 (127.28.244.129:34227): tablet f4d7a3603bcd4d0c9211ed7c3efe7c13 (table test-table [id=8cb05ad2dfb74fe2b0ba84ab5ca3675d]) successfully deleted
I20260929 20:28:58.810735 31906 raft_consensus.cc:482] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 69e7370ce6e948de9dd613dbe40af8a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:28:58.811028 31906 raft_consensus.cc:504] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 69e7370ce6e948de9dd613dbe40af8a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } }
I20260929 20:28:58.813154 31906 leader_election.cc:288] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 69e7370ce6e948de9dd613dbe40af8a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fd552a67209408b9d25ed9b4937d293 (127.28.244.129:34227), 218460624e544e9eb41d3cd524c8e9a7 (127.28.244.131:43369)
I20260929 20:28:58.832793 31951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:58.813998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:50471 (local address 127.28.244.129:34227)
0929 20:28:58.826332 (+ 12334us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:58.826338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:58.826356 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:28:58.826435 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:58.826439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:58.826513 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:58.826615 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:28:58.827454 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.828578 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.830507 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.831591 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.831787 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:58.831898 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:58.832279 (+ 381us) server_negotiation.cc:299] Negotiation successful
0929 20:28:58.832442 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12212,"thread_start_us":12110,"threads_started":1}
I20260929 20:28:58.832836 31950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:58.813373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:34227 (local address 127.28.244.130:50471)
0929 20:28:58.825453 (+ 12080us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:58.825487 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:58.825692 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:58.826713 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:58.826719 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:58.826776 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:28:58.827210 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:58.827225 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.828708 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.828719 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:58.830309 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:58.830324 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.831830 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.831837 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:58.832056 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:58.832102 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 20:28:58.832218 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 20:28:58.832408 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11943,"mutex_wait_us":1,"thread_start_us":12408,"threads_started":1}
I20260929 20:28:58.833983 31897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:58.813618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:49753 (local address 127.28.244.131:43369)
0929 20:28:58.813815 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:58.813822 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:58.825932 (+ 12110us) server_negotiation.cc:407] Connection header received
0929 20:28:58.826249 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:58.826255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:58.826335 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:58.826420 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:28:58.827160 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.828396 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.831053 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.832484 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.832644 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:58.832733 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:58.833538 (+ 805us) server_negotiation.cc:299] Negotiation successful
0929 20:28:58.833692 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 20:28:58.834002 31556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5b9a4eb9b0470ba655d5e55f8e9e7a" candidate_uuid: "69e7370ce6e948de9dd613dbe40af8a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fd552a67209408b9d25ed9b4937d293" is_pre_election: true
I20260929 20:28:58.834345 31556 raft_consensus.cc:2454] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 1fd552a67209408b9d25ed9b4937d293 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69e7370ce6e948de9dd613dbe40af8a7 in term 0.
I20260929 20:28:58.834789 31949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:58.813373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:43369 (local address 127.28.244.130:49753)
0929 20:28:58.825831 (+ 12458us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:58.825847 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:58.826039 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:58.826664 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:58.826669 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:58.826701 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:28:58.826972 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:58.826983 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.828581 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.828587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:58.830837 (+ 2250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:58.830854 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.832910 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.832919 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:58.833312 (+ 393us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:58.833355 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 20:28:58.834189 (+ 834us) client_negotiation.cc:240] Negotiation successful
0929 20:28:58.834354 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12319,"mutex_wait_us":148,"thread_start_us":11655,"threads_started":1}
I20260929 20:28:58.834951 31624 leader_election.cc:302] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 69e7370ce6e948de9dd613dbe40af8a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fd552a67209408b9d25ed9b4937d293, 69e7370ce6e948de9dd613dbe40af8a7; no voters:
I20260929 20:28:58.835476 31906 raft_consensus.cc:2781] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 69e7370ce6e948de9dd613dbe40af8a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:28:58.835613 31906 raft_consensus.cc:482] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 69e7370ce6e948de9dd613dbe40af8a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:28:58.835733 31906 raft_consensus.cc:3037] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 69e7370ce6e948de9dd613dbe40af8a7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:28:58.835855 31835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5b9a4eb9b0470ba655d5e55f8e9e7a" candidate_uuid: "69e7370ce6e948de9dd613dbe40af8a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218460624e544e9eb41d3cd524c8e9a7" is_pre_election: true
I20260929 20:28:58.836370 31835 raft_consensus.cc:2454] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 218460624e544e9eb41d3cd524c8e9a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69e7370ce6e948de9dd613dbe40af8a7 in term 0.
I20260929 20:28:58.838188 31906 raft_consensus.cc:504] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 69e7370ce6e948de9dd613dbe40af8a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } }
I20260929 20:28:58.838833 31906 leader_election.cc:288] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 69e7370ce6e948de9dd613dbe40af8a7 [CANDIDATE]: Term 1 election: Requested vote from peers 1fd552a67209408b9d25ed9b4937d293 (127.28.244.129:34227), 218460624e544e9eb41d3cd524c8e9a7 (127.28.244.131:43369)
I20260929 20:28:58.839413 31835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5b9a4eb9b0470ba655d5e55f8e9e7a" candidate_uuid: "69e7370ce6e948de9dd613dbe40af8a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218460624e544e9eb41d3cd524c8e9a7"
I20260929 20:28:58.839455 31556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5b9a4eb9b0470ba655d5e55f8e9e7a" candidate_uuid: "69e7370ce6e948de9dd613dbe40af8a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fd552a67209408b9d25ed9b4937d293"
I20260929 20:28:58.839665 31835 raft_consensus.cc:3037] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 218460624e544e9eb41d3cd524c8e9a7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:28:58.839725 31556 raft_consensus.cc:3037] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 1fd552a67209408b9d25ed9b4937d293 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:28:58.844887 31556 raft_consensus.cc:2454] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 1fd552a67209408b9d25ed9b4937d293 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69e7370ce6e948de9dd613dbe40af8a7 in term 1.
I20260929 20:28:58.845329 31835 raft_consensus.cc:2454] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 218460624e544e9eb41d3cd524c8e9a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69e7370ce6e948de9dd613dbe40af8a7 in term 1.
I20260929 20:28:58.845409 31624 leader_election.cc:302] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 69e7370ce6e948de9dd613dbe40af8a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fd552a67209408b9d25ed9b4937d293, 69e7370ce6e948de9dd613dbe40af8a7; no voters:
I20260929 20:28:58.846035 31906 raft_consensus.cc:2781] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 69e7370ce6e948de9dd613dbe40af8a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:28:58.846558 31906 raft_consensus.cc:686] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 69e7370ce6e948de9dd613dbe40af8a7 [term 1 LEADER]: Becoming Leader. State: Replica: 69e7370ce6e948de9dd613dbe40af8a7, State: Running, Role: LEADER
I20260929 20:28:58.847077 31906 consensus_queue.cc:237] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 69e7370ce6e948de9dd613dbe40af8a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } }
I20260929 20:28:58.852710 31395 catalog_manager.cc:5495] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 69e7370ce6e948de9dd613dbe40af8a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69e7370ce6e948de9dd613dbe40af8a7 (127.28.244.130). New cstate: current_term: 1 leader_uuid: "69e7370ce6e948de9dd613dbe40af8a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } health_report { overall_health: UNKNOWN } } }
I20260929 20:28:58.914808 31954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:58.908573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40484 (local address 127.28.244.130:46299)
0929 20:28:58.909202 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:58.909207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:58.909225 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:28:58.909308 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:58.909311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:58.909391 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:58.909572 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:28:58.910063 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.910821 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.912246 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.912697 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.912839 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:58.913300 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:28:58.914053 (+ 753us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:28:58.914212 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:58.914314 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 20:28:58.914403 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":313,"threads_started":1}
I20260929 20:28:58.933809 31897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:58.928048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:58859 (local address 127.28.244.131:43369)
0929 20:28:58.928273 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:58.928281 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:58.928738 (+ 457us) server_negotiation.cc:407] Connection header received
0929 20:28:58.929006 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:58.929010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:58.929079 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:58.929182 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:28:58.929710 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.930470 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.931948 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.932427 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.932562 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:58.933038 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:28:58.933148 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:28:58.933244 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:58.933387 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 20:28:58.933470 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:28:58.933912 31957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:58.927924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:43369 (local address 127.28.244.130:58859)
0929 20:28:58.928626 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:58.928642 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:58.928848 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:58.929214 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:58.929219 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:58.929231 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:28:58.929559 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:58.929569 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.930624 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.930631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:58.931736 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:58.931748 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.932620 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.932628 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:58.932779 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:58.932847 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:28:58.933246 (+ 399us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:28:58.933261 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 20:28:58.933365 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 20:28:58.933518 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":371,"threads_started":1}
I20260929 20:28:58.944425 31556 raft_consensus.cc:1261] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 1fd552a67209408b9d25ed9b4937d293 [term 1 FOLLOWER]: Refusing update from remote peer 69e7370ce6e948de9dd613dbe40af8a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:28:58.944977 31835 raft_consensus.cc:1261] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 218460624e544e9eb41d3cd524c8e9a7 [term 1 FOLLOWER]: Refusing update from remote peer 69e7370ce6e948de9dd613dbe40af8a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:28:58.945230 31952 consensus_queue.cc:1035] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 69e7370ce6e948de9dd613dbe40af8a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:28:58.945734 31906 consensus_queue.cc:1035] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 69e7370ce6e948de9dd613dbe40af8a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:28:58.954275 31927 mvcc.cc:205] Tried to move back new op lower bound from 7334763474708504576 to 7334763474323185664. Current Snapshot: MvccSnapshot[applied={T|T < 7334763474708504576}]
I20260929 20:28:58.988435 31954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:58.982269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:37387 (local address 127.28.244.130:46299)
0929 20:28:58.982457 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:58.982465 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:58.982801 (+ 336us) server_negotiation.cc:407] Connection header received
0929 20:28:58.983001 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:58.983006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:58.983108 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:58.983210 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:28:58.983892 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.984941 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.986481 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.986959 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.987098 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:58.987547 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:28:58.987701 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:28:58.987832 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:58.988031 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 20:28:58.988123 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 20:28:58.988490 31936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:58.982351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:46299 (local address 127.28.244.131:37387)
0929 20:28:58.982705 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:58.982725 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:58.982859 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:58.983421 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:58.983426 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:58.983440 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:28:58.983644 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:58.983661 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.985101 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.985107 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:58.986343 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:58.986360 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:58.987112 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:58.987117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:58.987286 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:58.987327 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:28:58.987899 (+ 572us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:28:58.987912 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 20:28:58.988001 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 20:28:58.988161 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186}
W20260929 20:28:58.990042 31927 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 20:29:00.614558 31395 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:29:00.614812 31395 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:29:00.619624 31395 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5884f7b08d4841ba82729ea2581cdd91: Sending DeleteTablet for 3 replicas of tablet bb5b9a4eb9b0470ba655d5e55f8e9e7a
I20260929 20:29:00.620626 31532 tablet_service.cc:1514] Processing DeleteTablet for tablet bb5b9a4eb9b0470ba655d5e55f8e9e7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:29:00 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:58434
I20260929 20:29:00.620642 31672 tablet_service.cc:1514] Processing DeleteTablet for tablet bb5b9a4eb9b0470ba655d5e55f8e9e7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:29:00 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:40472
I20260929 20:29:00.620736 31810 tablet_service.cc:1514] Processing DeleteTablet for tablet bb5b9a4eb9b0470ba655d5e55f8e9e7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:29:00 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:44320
I20260929 20:29:00.633008 31989 tablet_replica.cc:330] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 69e7370ce6e948de9dd613dbe40af8a7: stopping tablet replica
I20260929 20:29:00.633285 31989 raft_consensus.cc:2229] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 69e7370ce6e948de9dd613dbe40af8a7 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:29:00.633744 31990 tablet_replica.cc:330] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 218460624e544e9eb41d3cd524c8e9a7: stopping tablet replica
I20260929 20:29:00.633970 31989 raft_consensus.cc:2258] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 69e7370ce6e948de9dd613dbe40af8a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:29:00.634114 31990 raft_consensus.cc:2229] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 218460624e544e9eb41d3cd524c8e9a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:29:00.634436 31990 raft_consensus.cc:2258] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 218460624e544e9eb41d3cd524c8e9a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:29:00.634610 31988 tablet_replica.cc:330] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 1fd552a67209408b9d25ed9b4937d293: stopping tablet replica
I20260929 20:29:00.634812 31988 raft_consensus.cc:2229] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 1fd552a67209408b9d25ed9b4937d293 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:29:00.635077 31988 raft_consensus.cc:2258] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 1fd552a67209408b9d25ed9b4937d293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:29:00.635303 31989 ts_tablet_manager.cc:1905] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 69e7370ce6e948de9dd613dbe40af8a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:29:00.636116 31988 ts_tablet_manager.cc:1905] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 1fd552a67209408b9d25ed9b4937d293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:29:00.636113 31990 ts_tablet_manager.cc:1905] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 218460624e544e9eb41d3cd524c8e9a7: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 20:29:00 dist-test-slave-p29l krb5kdc[31350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713740, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260929 20:29:00.691638 31988 ts_tablet_manager.cc:1918] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 1fd552a67209408b9d25ed9b4937d293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 20:29:00.691869 31988 log.cc:1196] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 1fd552a67209408b9d25ed9b4937d293: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-0/wal/wals/bb5b9a4eb9b0470ba655d5e55f8e9e7a
I20260929 20:29:00.692340 31988 ts_tablet_manager.cc:1939] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 1fd552a67209408b9d25ed9b4937d293: Deleting consensus metadata
Sep 29 20:29:00 dist-test-slave-p29l krb5kdc[31350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713740, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:29:00.702384 31998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:00.686986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41756 (local address 127.28.244.190:37129)
0929 20:29:00.687650 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:00.687655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:00.687681 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:29:00.687907 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:00.687911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:00.687986 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:00.688088 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:00.689488 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:00.690369 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:00.692060 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:00.692548 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:00.692657 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:00.696802 (+ 4145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:00.696809 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:00.696813 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:00.696855 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:00.699700 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:00.700552 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:00.700557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:00.700560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:00.700714 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:00.701068 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:00.701072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:00.701074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:00.701357 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:00.701620 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:00.701842 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 20:29:00.702019 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":325,"threads_started":1}
I20260929 20:29:00.706672 31395 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:29:00.707259 31395 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:29:00.714733 31532 tablet_service.cc:1467] Processing CreateTablet for tablet ca70c667eb684eabb69c42b42edc5dec (DEFAULT_TABLE table=test-table [id=434ab3f6823341dbb10d9af50ac12fce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:29:00.714712 31672 tablet_service.cc:1467] Processing CreateTablet for tablet ca70c667eb684eabb69c42b42edc5dec (DEFAULT_TABLE table=test-table [id=434ab3f6823341dbb10d9af50ac12fce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:29:00.714922 31810 tablet_service.cc:1467] Processing CreateTablet for tablet ca70c667eb684eabb69c42b42edc5dec (DEFAULT_TABLE table=test-table [id=434ab3f6823341dbb10d9af50ac12fce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:29:00.715224 31532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca70c667eb684eabb69c42b42edc5dec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:00.721470 31999 tablet_bootstrap.cc:492] T ca70c667eb684eabb69c42b42edc5dec P 1fd552a67209408b9d25ed9b4937d293: Bootstrap starting.
I20260929 20:29:00.723933 31999 tablet_bootstrap.cc:654] T ca70c667eb684eabb69c42b42edc5dec P 1fd552a67209408b9d25ed9b4937d293: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:00.726387 31999 tablet_bootstrap.cc:492] T ca70c667eb684eabb69c42b42edc5dec P 1fd552a67209408b9d25ed9b4937d293: No bootstrap required, opened a new log
I20260929 20:29:00.726617 31999 ts_tablet_manager.cc:1397] T ca70c667eb684eabb69c42b42edc5dec P 1fd552a67209408b9d25ed9b4937d293: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 20:29:00.727329 31999 raft_consensus.cc:348] T ca70c667eb684eabb69c42b42edc5dec P 1fd552a67209408b9d25ed9b4937d293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } }
I20260929 20:29:00.727537 31999 raft_consensus.cc:374] T ca70c667eb684eabb69c42b42edc5dec P 1fd552a67209408b9d25ed9b4937d293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:00.727667 31999 raft_consensus.cc:729] T ca70c667eb684eabb69c42b42edc5dec P 1fd552a67209408b9d25ed9b4937d293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fd552a67209408b9d25ed9b4937d293, State: Initialized, Role: FOLLOWER
I20260929 20:29:00.727991 31999 consensus_queue.cc:260] T ca70c667eb684eabb69c42b42edc5dec P 1fd552a67209408b9d25ed9b4937d293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } }
I20260929 20:29:00.728820 31999 ts_tablet_manager.cc:1428] T ca70c667eb684eabb69c42b42edc5dec P 1fd552a67209408b9d25ed9b4937d293: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 20:29:00.769727 31381 catalog_manager.cc:4841] TS 1fd552a67209408b9d25ed9b4937d293 (127.28.244.129:34227): tablet bb5b9a4eb9b0470ba655d5e55f8e9e7a (table test-table [id=be919f9c0ab54b8f9e12b2cb98dc6624]) successfully deleted
I20260929 20:29:00.781962 31672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca70c667eb684eabb69c42b42edc5dec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:00.782043 31990 ts_tablet_manager.cc:1918] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 218460624e544e9eb41d3cd524c8e9a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 20:29:00.782047 31989 ts_tablet_manager.cc:1918] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 69e7370ce6e948de9dd613dbe40af8a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 20:29:00.782207 31990 log.cc:1196] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 218460624e544e9eb41d3cd524c8e9a7: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-2/wal/wals/bb5b9a4eb9b0470ba655d5e55f8e9e7a
I20260929 20:29:00.782207 31989 log.cc:1196] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 69e7370ce6e948de9dd613dbe40af8a7: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-1/wal/wals/bb5b9a4eb9b0470ba655d5e55f8e9e7a
I20260929 20:29:00.782743 31989 ts_tablet_manager.cc:1939] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 69e7370ce6e948de9dd613dbe40af8a7: Deleting consensus metadata
I20260929 20:29:00.782737 31990 ts_tablet_manager.cc:1939] T bb5b9a4eb9b0470ba655d5e55f8e9e7a P 218460624e544e9eb41d3cd524c8e9a7: Deleting consensus metadata
I20260929 20:29:00.783247 31810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca70c667eb684eabb69c42b42edc5dec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:00.789827 32002 tablet_bootstrap.cc:492] T ca70c667eb684eabb69c42b42edc5dec P 69e7370ce6e948de9dd613dbe40af8a7: Bootstrap starting.
I20260929 20:29:00.790707 32003 tablet_bootstrap.cc:492] T ca70c667eb684eabb69c42b42edc5dec P 218460624e544e9eb41d3cd524c8e9a7: Bootstrap starting.
I20260929 20:29:00.791499 32002 tablet_bootstrap.cc:654] T ca70c667eb684eabb69c42b42edc5dec P 69e7370ce6e948de9dd613dbe40af8a7: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:00.792301 32003 tablet_bootstrap.cc:654] T ca70c667eb684eabb69c42b42edc5dec P 218460624e544e9eb41d3cd524c8e9a7: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:00.794137 32002 tablet_bootstrap.cc:492] T ca70c667eb684eabb69c42b42edc5dec P 69e7370ce6e948de9dd613dbe40af8a7: No bootstrap required, opened a new log
I20260929 20:29:00.794312 32002 ts_tablet_manager.cc:1397] T ca70c667eb684eabb69c42b42edc5dec P 69e7370ce6e948de9dd613dbe40af8a7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 20:29:00.795410 32003 tablet_bootstrap.cc:492] T ca70c667eb684eabb69c42b42edc5dec P 218460624e544e9eb41d3cd524c8e9a7: No bootstrap required, opened a new log
I20260929 20:29:00.795254 32002 raft_consensus.cc:348] T ca70c667eb684eabb69c42b42edc5dec P 69e7370ce6e948de9dd613dbe40af8a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } }
I20260929 20:29:00.795542 32002 raft_consensus.cc:374] T ca70c667eb684eabb69c42b42edc5dec P 69e7370ce6e948de9dd613dbe40af8a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:00.795573 32003 ts_tablet_manager.cc:1397] T ca70c667eb684eabb69c42b42edc5dec P 218460624e544e9eb41d3cd524c8e9a7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 20:29:00.795639 32002 raft_consensus.cc:729] T ca70c667eb684eabb69c42b42edc5dec P 69e7370ce6e948de9dd613dbe40af8a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69e7370ce6e948de9dd613dbe40af8a7, State: Initialized, Role: FOLLOWER
I20260929 20:29:00.795688 31380 catalog_manager.cc:4841] TS 218460624e544e9eb41d3cd524c8e9a7 (127.28.244.131:43369): tablet bb5b9a4eb9b0470ba655d5e55f8e9e7a (table test-table [id=be919f9c0ab54b8f9e12b2cb98dc6624]) successfully deleted
I20260929 20:29:00.795967 31382 catalog_manager.cc:4841] TS 69e7370ce6e948de9dd613dbe40af8a7 (127.28.244.130:46299): tablet bb5b9a4eb9b0470ba655d5e55f8e9e7a (table test-table [id=be919f9c0ab54b8f9e12b2cb98dc6624]) successfully deleted
I20260929 20:29:00.795915 32002 consensus_queue.cc:260] T ca70c667eb684eabb69c42b42edc5dec P 69e7370ce6e948de9dd613dbe40af8a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } }
I20260929 20:29:00.796447 32002 ts_tablet_manager.cc:1428] T ca70c667eb684eabb69c42b42edc5dec P 69e7370ce6e948de9dd613dbe40af8a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 20:29:00.796450 32003 raft_consensus.cc:348] T ca70c667eb684eabb69c42b42edc5dec P 218460624e544e9eb41d3cd524c8e9a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } }
I20260929 20:29:00.796715 32003 raft_consensus.cc:374] T ca70c667eb684eabb69c42b42edc5dec P 218460624e544e9eb41d3cd524c8e9a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:00.796820 32003 raft_consensus.cc:729] T ca70c667eb684eabb69c42b42edc5dec P 218460624e544e9eb41d3cd524c8e9a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218460624e544e9eb41d3cd524c8e9a7, State: Initialized, Role: FOLLOWER
I20260929 20:29:00.797067 32003 consensus_queue.cc:260] T ca70c667eb684eabb69c42b42edc5dec P 218460624e544e9eb41d3cd524c8e9a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } }
I20260929 20:29:00.797652 32003 ts_tablet_manager.cc:1428] T ca70c667eb684eabb69c42b42edc5dec P 218460624e544e9eb41d3cd524c8e9a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 20:29:00.838877 31965 raft_consensus.cc:482] T ca70c667eb684eabb69c42b42edc5dec P 218460624e544e9eb41d3cd524c8e9a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:29:00.839107 31965 raft_consensus.cc:504] T ca70c667eb684eabb69c42b42edc5dec P 218460624e544e9eb41d3cd524c8e9a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } }
I20260929 20:29:00.840065 31965 leader_election.cc:288] T ca70c667eb684eabb69c42b42edc5dec P 218460624e544e9eb41d3cd524c8e9a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fd552a67209408b9d25ed9b4937d293 (127.28.244.129:34227), 69e7370ce6e948de9dd613dbe40af8a7 (127.28.244.130:46299)
I20260929 20:29:00.840675 31557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca70c667eb684eabb69c42b42edc5dec" candidate_uuid: "218460624e544e9eb41d3cd524c8e9a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fd552a67209408b9d25ed9b4937d293" is_pre_election: true
I20260929 20:29:00.840773 31691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca70c667eb684eabb69c42b42edc5dec" candidate_uuid: "218460624e544e9eb41d3cd524c8e9a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69e7370ce6e948de9dd613dbe40af8a7" is_pre_election: true
I20260929 20:29:00.840965 31557 raft_consensus.cc:2454] T ca70c667eb684eabb69c42b42edc5dec P 1fd552a67209408b9d25ed9b4937d293 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 218460624e544e9eb41d3cd524c8e9a7 in term 0.
I20260929 20:29:00.841118 31691 raft_consensus.cc:2454] T ca70c667eb684eabb69c42b42edc5dec P 69e7370ce6e948de9dd613dbe40af8a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 218460624e544e9eb41d3cd524c8e9a7 in term 0.
I20260929 20:29:00.841507 31764 leader_election.cc:302] T ca70c667eb684eabb69c42b42edc5dec P 218460624e544e9eb41d3cd524c8e9a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fd552a67209408b9d25ed9b4937d293, 218460624e544e9eb41d3cd524c8e9a7; no voters:
I20260929 20:29:00.841785 31965 raft_consensus.cc:2781] T ca70c667eb684eabb69c42b42edc5dec P 218460624e544e9eb41d3cd524c8e9a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:29:00.841918 31965 raft_consensus.cc:482] T ca70c667eb684eabb69c42b42edc5dec P 218460624e544e9eb41d3cd524c8e9a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:29:00.842034 31965 raft_consensus.cc:3037] T ca70c667eb684eabb69c42b42edc5dec P 218460624e544e9eb41d3cd524c8e9a7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:00.877910 31965 raft_consensus.cc:504] T ca70c667eb684eabb69c42b42edc5dec P 218460624e544e9eb41d3cd524c8e9a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } }
I20260929 20:29:00.878973 31965 leader_election.cc:288] T ca70c667eb684eabb69c42b42edc5dec P 218460624e544e9eb41d3cd524c8e9a7 [CANDIDATE]: Term 1 election: Requested vote from peers 1fd552a67209408b9d25ed9b4937d293 (127.28.244.129:34227), 69e7370ce6e948de9dd613dbe40af8a7 (127.28.244.130:46299)
I20260929 20:29:00.879396 31557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca70c667eb684eabb69c42b42edc5dec" candidate_uuid: "218460624e544e9eb41d3cd524c8e9a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fd552a67209408b9d25ed9b4937d293"
I20260929 20:29:00.879406 31691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca70c667eb684eabb69c42b42edc5dec" candidate_uuid: "218460624e544e9eb41d3cd524c8e9a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69e7370ce6e948de9dd613dbe40af8a7"
I20260929 20:29:00.879629 31691 raft_consensus.cc:3037] T ca70c667eb684eabb69c42b42edc5dec P 69e7370ce6e948de9dd613dbe40af8a7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:00.879676 31557 raft_consensus.cc:3037] T ca70c667eb684eabb69c42b42edc5dec P 1fd552a67209408b9d25ed9b4937d293 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:00.929549 31557 raft_consensus.cc:2454] T ca70c667eb684eabb69c42b42edc5dec P 1fd552a67209408b9d25ed9b4937d293 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 218460624e544e9eb41d3cd524c8e9a7 in term 1.
I20260929 20:29:00.929587 31691 raft_consensus.cc:2454] T ca70c667eb684eabb69c42b42edc5dec P 69e7370ce6e948de9dd613dbe40af8a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 218460624e544e9eb41d3cd524c8e9a7 in term 1.
I20260929 20:29:00.930233 31764 leader_election.cc:302] T ca70c667eb684eabb69c42b42edc5dec P 218460624e544e9eb41d3cd524c8e9a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fd552a67209408b9d25ed9b4937d293, 218460624e544e9eb41d3cd524c8e9a7; no voters:
I20260929 20:29:00.930785 31965 raft_consensus.cc:2781] T ca70c667eb684eabb69c42b42edc5dec P 218460624e544e9eb41d3cd524c8e9a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:29:00.930981 31965 raft_consensus.cc:686] T ca70c667eb684eabb69c42b42edc5dec P 218460624e544e9eb41d3cd524c8e9a7 [term 1 LEADER]: Becoming Leader. State: Replica: 218460624e544e9eb41d3cd524c8e9a7, State: Running, Role: LEADER
I20260929 20:29:00.931286 31965 consensus_queue.cc:237] T ca70c667eb684eabb69c42b42edc5dec P 218460624e544e9eb41d3cd524c8e9a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } }
I20260929 20:29:00.934362 31393 catalog_manager.cc:5495] T ca70c667eb684eabb69c42b42edc5dec P 218460624e544e9eb41d3cd524c8e9a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 218460624e544e9eb41d3cd524c8e9a7 (127.28.244.131). New cstate: current_term: 1 leader_uuid: "218460624e544e9eb41d3cd524c8e9a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } health_report { overall_health: HEALTHY } } }
I20260929 20:29:00.977442 32008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:00.972278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44348 (local address 127.28.244.131:43369)
0929 20:29:00.972866 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:00.972871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:00.972890 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:29:00.972944 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:00.972948 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:00.973023 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:00.973105 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:29:00.973466 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:00.974477 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:00.975752 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:00.976241 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:00.976338 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:00.976731 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:29:00.976828 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:29:00.976894 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:00.977024 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 20:29:00.977085 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":356,"threads_started":1}
I20260929 20:29:00.986970 31557 raft_consensus.cc:1261] T ca70c667eb684eabb69c42b42edc5dec P 1fd552a67209408b9d25ed9b4937d293 [term 1 FOLLOWER]: Refusing update from remote peer 218460624e544e9eb41d3cd524c8e9a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:29:00.986953 31691 raft_consensus.cc:1261] T ca70c667eb684eabb69c42b42edc5dec P 69e7370ce6e948de9dd613dbe40af8a7 [term 1 FOLLOWER]: Refusing update from remote peer 218460624e544e9eb41d3cd524c8e9a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:29:00.987492 31969 consensus_queue.cc:1035] T ca70c667eb684eabb69c42b42edc5dec P 218460624e544e9eb41d3cd524c8e9a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:29:00.987923 31983 consensus_queue.cc:1035] T ca70c667eb684eabb69c42b42edc5dec P 218460624e544e9eb41d3cd524c8e9a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:29:01.038379 31395 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:29:01.038633 31395 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:29:01.041878 31395 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5884f7b08d4841ba82729ea2581cdd91: Sending DeleteTablet for 3 replicas of tablet ca70c667eb684eabb69c42b42edc5dec
I20260929 20:29:01.042951 31811 tablet_service.cc:1514] Processing DeleteTablet for tablet ca70c667eb684eabb69c42b42edc5dec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:29:01 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:44320
I20260929 20:29:01.042954 31532 tablet_service.cc:1514] Processing DeleteTablet for tablet ca70c667eb684eabb69c42b42edc5dec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:29:01 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:58434
I20260929 20:29:01.043277 31990 tablet_replica.cc:330] T ca70c667eb684eabb69c42b42edc5dec P 218460624e544e9eb41d3cd524c8e9a7: stopping tablet replica
I20260929 20:29:01.043303 31672 tablet_service.cc:1514] Processing DeleteTablet for tablet ca70c667eb684eabb69c42b42edc5dec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:29:01 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:40472
I20260929 20:29:01.043503 31990 raft_consensus.cc:2229] T ca70c667eb684eabb69c42b42edc5dec P 218460624e544e9eb41d3cd524c8e9a7 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:29:01.043615 31989 tablet_replica.cc:330] T ca70c667eb684eabb69c42b42edc5dec P 69e7370ce6e948de9dd613dbe40af8a7: stopping tablet replica
I20260929 20:29:01.043704 31988 tablet_replica.cc:330] T ca70c667eb684eabb69c42b42edc5dec P 1fd552a67209408b9d25ed9b4937d293: stopping tablet replica
I20260929 20:29:01.043862 31989 raft_consensus.cc:2229] T ca70c667eb684eabb69c42b42edc5dec P 69e7370ce6e948de9dd613dbe40af8a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:29:01.043924 31990 raft_consensus.cc:2258] T ca70c667eb684eabb69c42b42edc5dec P 218460624e544e9eb41d3cd524c8e9a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:29:01.043949 31988 raft_consensus.cc:2229] T ca70c667eb684eabb69c42b42edc5dec P 1fd552a67209408b9d25ed9b4937d293 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:29:01.044076 31989 raft_consensus.cc:2258] T ca70c667eb684eabb69c42b42edc5dec P 69e7370ce6e948de9dd613dbe40af8a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:29:01.044143 31988 raft_consensus.cc:2258] T ca70c667eb684eabb69c42b42edc5dec P 1fd552a67209408b9d25ed9b4937d293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:29:01.045142 31990 ts_tablet_manager.cc:1905] T ca70c667eb684eabb69c42b42edc5dec P 218460624e544e9eb41d3cd524c8e9a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:29:01.045187 31988 ts_tablet_manager.cc:1905] T ca70c667eb684eabb69c42b42edc5dec P 1fd552a67209408b9d25ed9b4937d293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:29:01.045228 31989 ts_tablet_manager.cc:1905] T ca70c667eb684eabb69c42b42edc5dec P 69e7370ce6e948de9dd613dbe40af8a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:29:01.045564 31395 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:29:01.046269 31395 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:29:01.053582 31532 tablet_service.cc:1467] Processing CreateTablet for tablet 1208c7708e3f4c88a77344f79d710234 (DEFAULT_TABLE table=test-table [id=e30c5b8ebd3641a08413edd7bb9b3dcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:29:01.053838 31672 tablet_service.cc:1467] Processing CreateTablet for tablet 1208c7708e3f4c88a77344f79d710234 (DEFAULT_TABLE table=test-table [id=e30c5b8ebd3641a08413edd7bb9b3dcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:29:01.053954 31811 tablet_service.cc:1467] Processing CreateTablet for tablet 1208c7708e3f4c88a77344f79d710234 (DEFAULT_TABLE table=test-table [id=e30c5b8ebd3641a08413edd7bb9b3dcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:29:01.130059 31811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1208c7708e3f4c88a77344f79d710234. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:01.130128 31532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1208c7708e3f4c88a77344f79d710234. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:01.134227 31990 ts_tablet_manager.cc:1918] T ca70c667eb684eabb69c42b42edc5dec P 218460624e544e9eb41d3cd524c8e9a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:29:01.134464 31990 log.cc:1196] T ca70c667eb684eabb69c42b42edc5dec P 218460624e544e9eb41d3cd524c8e9a7: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-2/wal/wals/ca70c667eb684eabb69c42b42edc5dec
I20260929 20:29:01.134963 31990 ts_tablet_manager.cc:1939] T ca70c667eb684eabb69c42b42edc5dec P 218460624e544e9eb41d3cd524c8e9a7: Deleting consensus metadata
I20260929 20:29:01.137107 31672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1208c7708e3f4c88a77344f79d710234. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:01.137962 32003 tablet_bootstrap.cc:492] T 1208c7708e3f4c88a77344f79d710234 P 218460624e544e9eb41d3cd524c8e9a7: Bootstrap starting.
I20260929 20:29:01.139689 32003 tablet_bootstrap.cc:654] T 1208c7708e3f4c88a77344f79d710234 P 218460624e544e9eb41d3cd524c8e9a7: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:01.139892 31999 tablet_bootstrap.cc:492] T 1208c7708e3f4c88a77344f79d710234 P 1fd552a67209408b9d25ed9b4937d293: Bootstrap starting.
I20260929 20:29:01.141654 31999 tablet_bootstrap.cc:654] T 1208c7708e3f4c88a77344f79d710234 P 1fd552a67209408b9d25ed9b4937d293: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:01.146178 32002 tablet_bootstrap.cc:492] T 1208c7708e3f4c88a77344f79d710234 P 69e7370ce6e948de9dd613dbe40af8a7: Bootstrap starting.
I20260929 20:29:01.147213 32003 tablet_bootstrap.cc:492] T 1208c7708e3f4c88a77344f79d710234 P 218460624e544e9eb41d3cd524c8e9a7: No bootstrap required, opened a new log
I20260929 20:29:01.147264 31999 tablet_bootstrap.cc:492] T 1208c7708e3f4c88a77344f79d710234 P 1fd552a67209408b9d25ed9b4937d293: No bootstrap required, opened a new log
I20260929 20:29:01.147433 31999 ts_tablet_manager.cc:1397] T 1208c7708e3f4c88a77344f79d710234 P 1fd552a67209408b9d25ed9b4937d293: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 20:29:01.147491 32003 ts_tablet_manager.cc:1397] T 1208c7708e3f4c88a77344f79d710234 P 218460624e544e9eb41d3cd524c8e9a7: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260929 20:29:01.148066 32002 tablet_bootstrap.cc:654] T 1208c7708e3f4c88a77344f79d710234 P 69e7370ce6e948de9dd613dbe40af8a7: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:01.148288 32003 raft_consensus.cc:348] T 1208c7708e3f4c88a77344f79d710234 P 218460624e544e9eb41d3cd524c8e9a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } }
I20260929 20:29:01.148316 31999 raft_consensus.cc:348] T 1208c7708e3f4c88a77344f79d710234 P 1fd552a67209408b9d25ed9b4937d293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } }
I20260929 20:29:01.148531 32003 raft_consensus.cc:374] T 1208c7708e3f4c88a77344f79d710234 P 218460624e544e9eb41d3cd524c8e9a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:01.148588 32003 raft_consensus.cc:729] T 1208c7708e3f4c88a77344f79d710234 P 218460624e544e9eb41d3cd524c8e9a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218460624e544e9eb41d3cd524c8e9a7, State: Initialized, Role: FOLLOWER
I20260929 20:29:01.148589 31999 raft_consensus.cc:374] T 1208c7708e3f4c88a77344f79d710234 P 1fd552a67209408b9d25ed9b4937d293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:01.148672 31999 raft_consensus.cc:729] T 1208c7708e3f4c88a77344f79d710234 P 1fd552a67209408b9d25ed9b4937d293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fd552a67209408b9d25ed9b4937d293, State: Initialized, Role: FOLLOWER
I20260929 20:29:01.148885 32003 consensus_queue.cc:260] T 1208c7708e3f4c88a77344f79d710234 P 218460624e544e9eb41d3cd524c8e9a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } }
I20260929 20:29:01.148909 31999 consensus_queue.cc:260] T 1208c7708e3f4c88a77344f79d710234 P 1fd552a67209408b9d25ed9b4937d293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } }
I20260929 20:29:01.149652 31999 ts_tablet_manager.cc:1428] T 1208c7708e3f4c88a77344f79d710234 P 1fd552a67209408b9d25ed9b4937d293: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 20:29:01.150319 32003 ts_tablet_manager.cc:1428] T 1208c7708e3f4c88a77344f79d710234 P 218460624e544e9eb41d3cd524c8e9a7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 20:29:01.150532 32002 tablet_bootstrap.cc:492] T 1208c7708e3f4c88a77344f79d710234 P 69e7370ce6e948de9dd613dbe40af8a7: No bootstrap required, opened a new log
I20260929 20:29:01.150715 32002 ts_tablet_manager.cc:1397] T 1208c7708e3f4c88a77344f79d710234 P 69e7370ce6e948de9dd613dbe40af8a7: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260929 20:29:01.151526 32002 raft_consensus.cc:348] T 1208c7708e3f4c88a77344f79d710234 P 69e7370ce6e948de9dd613dbe40af8a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } }
I20260929 20:29:01.151786 32002 raft_consensus.cc:374] T 1208c7708e3f4c88a77344f79d710234 P 69e7370ce6e948de9dd613dbe40af8a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:01.151913 32002 raft_consensus.cc:729] T 1208c7708e3f4c88a77344f79d710234 P 69e7370ce6e948de9dd613dbe40af8a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69e7370ce6e948de9dd613dbe40af8a7, State: Initialized, Role: FOLLOWER
I20260929 20:29:01.152244 32002 consensus_queue.cc:260] T 1208c7708e3f4c88a77344f79d710234 P 69e7370ce6e948de9dd613dbe40af8a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } }
I20260929 20:29:01.152856 32002 ts_tablet_manager.cc:1428] T 1208c7708e3f4c88a77344f79d710234 P 69e7370ce6e948de9dd613dbe40af8a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 20:29:01.181310 31966 raft_consensus.cc:482] T 1208c7708e3f4c88a77344f79d710234 P 69e7370ce6e948de9dd613dbe40af8a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:29:01.181691 31966 raft_consensus.cc:504] T 1208c7708e3f4c88a77344f79d710234 P 69e7370ce6e948de9dd613dbe40af8a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } }
I20260929 20:29:01.182601 31966 leader_election.cc:288] T 1208c7708e3f4c88a77344f79d710234 P 69e7370ce6e948de9dd613dbe40af8a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fd552a67209408b9d25ed9b4937d293 (127.28.244.129:34227), 218460624e544e9eb41d3cd524c8e9a7 (127.28.244.131:43369)
I20260929 20:29:01.183266 31557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1208c7708e3f4c88a77344f79d710234" candidate_uuid: "69e7370ce6e948de9dd613dbe40af8a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fd552a67209408b9d25ed9b4937d293" is_pre_election: true
I20260929 20:29:01.183394 31835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1208c7708e3f4c88a77344f79d710234" candidate_uuid: "69e7370ce6e948de9dd613dbe40af8a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218460624e544e9eb41d3cd524c8e9a7" is_pre_election: true
I20260929 20:29:01.183576 31557 raft_consensus.cc:2454] T 1208c7708e3f4c88a77344f79d710234 P 1fd552a67209408b9d25ed9b4937d293 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69e7370ce6e948de9dd613dbe40af8a7 in term 0.
I20260929 20:29:01.183705 31835 raft_consensus.cc:2454] T 1208c7708e3f4c88a77344f79d710234 P 218460624e544e9eb41d3cd524c8e9a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69e7370ce6e948de9dd613dbe40af8a7 in term 0.
I20260929 20:29:01.184077 31624 leader_election.cc:302] T 1208c7708e3f4c88a77344f79d710234 P 69e7370ce6e948de9dd613dbe40af8a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fd552a67209408b9d25ed9b4937d293, 69e7370ce6e948de9dd613dbe40af8a7; no voters:
I20260929 20:29:01.184352 31966 raft_consensus.cc:2781] T 1208c7708e3f4c88a77344f79d710234 P 69e7370ce6e948de9dd613dbe40af8a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:29:01.184504 31966 raft_consensus.cc:482] T 1208c7708e3f4c88a77344f79d710234 P 69e7370ce6e948de9dd613dbe40af8a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:29:01.184631 31966 raft_consensus.cc:3037] T 1208c7708e3f4c88a77344f79d710234 P 69e7370ce6e948de9dd613dbe40af8a7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:01.190572 31380 catalog_manager.cc:4841] TS 218460624e544e9eb41d3cd524c8e9a7 (127.28.244.131:43369): tablet ca70c667eb684eabb69c42b42edc5dec (table test-table [id=434ab3f6823341dbb10d9af50ac12fce]) successfully deleted
I20260929 20:29:01.229056 31988 ts_tablet_manager.cc:1918] T ca70c667eb684eabb69c42b42edc5dec P 1fd552a67209408b9d25ed9b4937d293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:29:01.229060 31989 ts_tablet_manager.cc:1918] T ca70c667eb684eabb69c42b42edc5dec P 69e7370ce6e948de9dd613dbe40af8a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:29:01.228999 31966 raft_consensus.cc:504] T 1208c7708e3f4c88a77344f79d710234 P 69e7370ce6e948de9dd613dbe40af8a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } }
I20260929 20:29:01.229265 31988 log.cc:1196] T ca70c667eb684eabb69c42b42edc5dec P 1fd552a67209408b9d25ed9b4937d293: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-0/wal/wals/ca70c667eb684eabb69c42b42edc5dec
I20260929 20:29:01.229271 31989 log.cc:1196] T ca70c667eb684eabb69c42b42edc5dec P 69e7370ce6e948de9dd613dbe40af8a7: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-1/wal/wals/ca70c667eb684eabb69c42b42edc5dec
I20260929 20:29:01.229782 31988 ts_tablet_manager.cc:1939] T ca70c667eb684eabb69c42b42edc5dec P 1fd552a67209408b9d25ed9b4937d293: Deleting consensus metadata
I20260929 20:29:01.229795 31989 ts_tablet_manager.cc:1939] T ca70c667eb684eabb69c42b42edc5dec P 69e7370ce6e948de9dd613dbe40af8a7: Deleting consensus metadata
I20260929 20:29:01.229957 31966 leader_election.cc:288] T 1208c7708e3f4c88a77344f79d710234 P 69e7370ce6e948de9dd613dbe40af8a7 [CANDIDATE]: Term 1 election: Requested vote from peers 1fd552a67209408b9d25ed9b4937d293 (127.28.244.129:34227), 218460624e544e9eb41d3cd524c8e9a7 (127.28.244.131:43369)
I20260929 20:29:01.231123 31381 catalog_manager.cc:4841] TS 1fd552a67209408b9d25ed9b4937d293 (127.28.244.129:34227): tablet ca70c667eb684eabb69c42b42edc5dec (table test-table [id=434ab3f6823341dbb10d9af50ac12fce]) successfully deleted
I20260929 20:29:01.231240 31382 catalog_manager.cc:4841] TS 69e7370ce6e948de9dd613dbe40af8a7 (127.28.244.130:46299): tablet ca70c667eb684eabb69c42b42edc5dec (table test-table [id=434ab3f6823341dbb10d9af50ac12fce]) successfully deleted
I20260929 20:29:01.231458 31835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1208c7708e3f4c88a77344f79d710234" candidate_uuid: "69e7370ce6e948de9dd613dbe40af8a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218460624e544e9eb41d3cd524c8e9a7"
I20260929 20:29:01.231544 31557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1208c7708e3f4c88a77344f79d710234" candidate_uuid: "69e7370ce6e948de9dd613dbe40af8a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fd552a67209408b9d25ed9b4937d293"
I20260929 20:29:01.231738 31835 raft_consensus.cc:3037] T 1208c7708e3f4c88a77344f79d710234 P 218460624e544e9eb41d3cd524c8e9a7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:01.231760 31557 raft_consensus.cc:3037] T 1208c7708e3f4c88a77344f79d710234 P 1fd552a67209408b9d25ed9b4937d293 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:01.277159 31557 raft_consensus.cc:2454] T 1208c7708e3f4c88a77344f79d710234 P 1fd552a67209408b9d25ed9b4937d293 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69e7370ce6e948de9dd613dbe40af8a7 in term 1.
I20260929 20:29:01.277182 31835 raft_consensus.cc:2454] T 1208c7708e3f4c88a77344f79d710234 P 218460624e544e9eb41d3cd524c8e9a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69e7370ce6e948de9dd613dbe40af8a7 in term 1.
I20260929 20:29:01.277878 31623 leader_election.cc:302] T 1208c7708e3f4c88a77344f79d710234 P 69e7370ce6e948de9dd613dbe40af8a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 218460624e544e9eb41d3cd524c8e9a7, 69e7370ce6e948de9dd613dbe40af8a7; no voters:
I20260929 20:29:01.278343 31966 raft_consensus.cc:2781] T 1208c7708e3f4c88a77344f79d710234 P 69e7370ce6e948de9dd613dbe40af8a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:29:01.278820 31966 raft_consensus.cc:686] T 1208c7708e3f4c88a77344f79d710234 P 69e7370ce6e948de9dd613dbe40af8a7 [term 1 LEADER]: Becoming Leader. State: Replica: 69e7370ce6e948de9dd613dbe40af8a7, State: Running, Role: LEADER
I20260929 20:29:01.279148 31966 consensus_queue.cc:237] T 1208c7708e3f4c88a77344f79d710234 P 69e7370ce6e948de9dd613dbe40af8a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } }
I20260929 20:29:01.282780 31394 catalog_manager.cc:5495] T 1208c7708e3f4c88a77344f79d710234 P 69e7370ce6e948de9dd613dbe40af8a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69e7370ce6e948de9dd613dbe40af8a7 (127.28.244.130). New cstate: current_term: 1 leader_uuid: "69e7370ce6e948de9dd613dbe40af8a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69e7370ce6e948de9dd613dbe40af8a7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 46299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 } health_report { overall_health: UNKNOWN } } }
I20260929 20:29:01.318190 32017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:01.312175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40500 (local address 127.28.244.130:46299)
0929 20:29:01.312912 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:01.312917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:01.312938 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:29:01.312997 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:01.313001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:01.313067 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:01.313190 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:29:01.313667 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:01.314557 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:01.316095 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:01.316566 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:01.316720 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:01.317234 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:29:01.317369 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:29:01.317489 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:01.317709 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 20:29:01.317803 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":347,"threads_started":1}
I20260929 20:29:01.329360 31835 raft_consensus.cc:1261] T 1208c7708e3f4c88a77344f79d710234 P 218460624e544e9eb41d3cd524c8e9a7 [term 1 FOLLOWER]: Refusing update from remote peer 69e7370ce6e948de9dd613dbe40af8a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:29:01.329418 31557 raft_consensus.cc:1261] T 1208c7708e3f4c88a77344f79d710234 P 1fd552a67209408b9d25ed9b4937d293 [term 1 FOLLOWER]: Refusing update from remote peer 69e7370ce6e948de9dd613dbe40af8a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:29:01.330031 31966 consensus_queue.cc:1035] T 1208c7708e3f4c88a77344f79d710234 P 69e7370ce6e948de9dd613dbe40af8a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fd552a67209408b9d25ed9b4937d293" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:29:01.330335 32015 consensus_queue.cc:1035] T 1208c7708e3f4c88a77344f79d710234 P 69e7370ce6e948de9dd613dbe40af8a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218460624e544e9eb41d3cd524c8e9a7" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 20:29:01.350440 31970 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 20:29:02.952905 31395 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:29:02.953219 31395 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:29:02.960090 31395 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5884f7b08d4841ba82729ea2581cdd91: Sending DeleteTablet for 3 replicas of tablet 1208c7708e3f4c88a77344f79d710234
I20260929 20:29:02.961196 31532 tablet_service.cc:1514] Processing DeleteTablet for tablet 1208c7708e3f4c88a77344f79d710234 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:29:02 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:58434
I20260929 20:29:02.961246 31811 tablet_service.cc:1514] Processing DeleteTablet for tablet 1208c7708e3f4c88a77344f79d710234 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:29:02 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:44320
I20260929 20:29:02.962092 32043 tablet_replica.cc:330] T 1208c7708e3f4c88a77344f79d710234 P 1fd552a67209408b9d25ed9b4937d293: stopping tablet replica
I20260929 20:29:02.962360 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 31466
I20260929 20:29:02.962550 32043 raft_consensus.cc:2229] T 1208c7708e3f4c88a77344f79d710234 P 1fd552a67209408b9d25ed9b4937d293 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:29:02.962852 32043 raft_consensus.cc:2258] T 1208c7708e3f4c88a77344f79d710234 P 1fd552a67209408b9d25ed9b4937d293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:29:02.964275 32043 ts_tablet_manager.cc:1905] T 1208c7708e3f4c88a77344f79d710234 P 1fd552a67209408b9d25ed9b4937d293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:29:02.965667 31672 tablet_service.cc:1514] Processing DeleteTablet for tablet 1208c7708e3f4c88a77344f79d710234 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:29:02 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:40472
I20260929 20:29:02.968699 32042 tablet_replica.cc:330] T 1208c7708e3f4c88a77344f79d710234 P 218460624e544e9eb41d3cd524c8e9a7: stopping tablet replica
I20260929 20:29:02.969017 32042 raft_consensus.cc:2229] T 1208c7708e3f4c88a77344f79d710234 P 218460624e544e9eb41d3cd524c8e9a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:29:02.969321 32042 raft_consensus.cc:2258] T 1208c7708e3f4c88a77344f79d710234 P 218460624e544e9eb41d3cd524c8e9a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:29:02.971586 32042 ts_tablet_manager.cc:1905] T 1208c7708e3f4c88a77344f79d710234 P 218460624e544e9eb41d3cd524c8e9a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:29:02.977275 32046 tablet_replica.cc:330] T 1208c7708e3f4c88a77344f79d710234 P 69e7370ce6e948de9dd613dbe40af8a7: stopping tablet replica
I20260929 20:29:02.977562 32046 raft_consensus.cc:2229] T 1208c7708e3f4c88a77344f79d710234 P 69e7370ce6e948de9dd613dbe40af8a7 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:29:02.978055 32046 raft_consensus.cc:2258] T 1208c7708e3f4c88a77344f79d710234 P 69e7370ce6e948de9dd613dbe40af8a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:29:02.979713 32046 ts_tablet_manager.cc:1905] T 1208c7708e3f4c88a77344f79d710234 P 69e7370ce6e948de9dd613dbe40af8a7: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 20:29:02 dist-test-slave-p29l krb5kdc[31350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713740, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.244.129@KRBTEST.COM
I20260929 20:29:03.002053 32044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:02.963505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58450 (local address 127.28.244.129:34227)
0929 20:29:02.980349 (+ 16844us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:02.980355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:02.980377 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:29:02.980446 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:02.980450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:02.980543 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:02.982744 (+ 2201us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:02.982961 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:02.984315 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:02.986771 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:02.987521 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:02.987682 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:02.993356 (+ 5674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:02.993380 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:02.993405 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:02.993458 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:02.997763 (+ 4305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:02.998894 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:02.998921 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:02.998926 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:02.999129 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:02.999752 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:02.999762 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:02.999768 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:03.000434 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:03.000857 (+ 423us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:03.001077 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 20:29:03.001512 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16687,"thread_start_us":16557,"threads_started":1}
W20260929 20:29:03.002923 31597 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58450
W20260929 20:29:03.003758 29650 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 31466: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:29:03.004117 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 31466
I20260929 20:29:03.042322 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 31606
W20260929 20:29:03.042920 31381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.244.129:34227 (error 108)
I20260929 20:29:03.045782 32049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:03.044654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:34227 (local address 127.0.0.1:58458)
0929 20:29:03.045293 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:03.045506 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:34227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":308,"threads_started":1}
W20260929 20:29:03.046141 31381 catalog_manager.cc:4584] TS 1fd552a67209408b9d25ed9b4937d293 (127.28.244.129:34227): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1208c7708e3f4c88a77344f79d710234: Network error: Client connection negotiation failed: client connection to 127.28.244.129:34227: connect: Connection refused (error 111)
Sep 29 20:29:03 dist-test-slave-p29l krb5kdc[31350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713740, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.244.130@KRBTEST.COM
I20260929 20:29:03.060593 32047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:03.043220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40516 (local address 127.28.244.130:46299)
0929 20:29:03.043892 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:03.043899 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:03.043925 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:29:03.044008 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:03.044013 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:03.044149 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:03.044281 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:03.045725 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:03.046787 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:03.048306 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:03.048857 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:03.048993 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:03.053686 (+ 4693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:03.053707 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:03.053725 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:03.053772 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:03.057053 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:03.058121 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:03.058140 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:03.058142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:03.058304 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:03.058788 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:03.058792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:03.058794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:03.059297 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:03.059639 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:03.060075 (+ 436us) server_negotiation.cc:299] Negotiation successful
0929 20:29:03.060282 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":282,"threads_started":1}
W20260929 20:29:03.061412 31737 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40516
W20260929 20:29:03.062083 29650 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 31606: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:29:03.062222 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 31606
I20260929 20:29:03.100148 32049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:03.099558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:34227 (local address 127.0.0.1:58474)
0929 20:29:03.099860 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:03.099979 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:34227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260929 20:29:03.101380 32049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:03.100902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:34227 (local address 127.0.0.1:58476)
0929 20:29:03.101080 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:03.101237 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:34227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 20:29:03.111899 32042 ts_tablet_manager.cc:1918] T 1208c7708e3f4c88a77344f79d710234 P 218460624e544e9eb41d3cd524c8e9a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 20:29:03.112115 32042 log.cc:1196] T 1208c7708e3f4c88a77344f79d710234 P 218460624e544e9eb41d3cd524c8e9a7: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713721201454-29650-0/minicluster-data/ts-2/wal/wals/1208c7708e3f4c88a77344f79d710234
I20260929 20:29:03.112679 32042 ts_tablet_manager.cc:1939] T 1208c7708e3f4c88a77344f79d710234 P 218460624e544e9eb41d3cd524c8e9a7: Deleting consensus metadata
I20260929 20:29:03.114233 31380 catalog_manager.cc:4841] TS 218460624e544e9eb41d3cd524c8e9a7 (127.28.244.131:43369): tablet 1208c7708e3f4c88a77344f79d710234 (table test-table [id=e30c5b8ebd3641a08413edd7bb9b3dcf]) successfully deleted
I20260929 20:29:03.135318 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 31746
I20260929 20:29:03.136475 32049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:03.135951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:46299 (local address 127.0.0.1:40526)
0929 20:29:03.136167 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:03.136279 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.130:46299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
Sep 29 20:29:03 dist-test-slave-p29l krb5kdc[31350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713740, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.244.131@KRBTEST.COM
I20260929 20:29:03.154925 32050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:03.136284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44354 (local address 127.28.244.131:43369)
0929 20:29:03.136911 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:03.136917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:03.136933 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:29:03.136989 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:03.136993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:03.137064 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:03.137206 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:03.138739 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:03.139565 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:03.141081 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:03.141596 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:03.141747 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:03.147798 (+ 6051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:03.147873 (+ 75us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:03.147906 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:03.147971 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:03.151359 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:03.152434 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:03.152444 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:03.152447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:03.152618 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:03.153092 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:03.153097 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:03.153099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:03.153524 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:03.153879 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:03.154259 (+ 380us) server_negotiation.cc:299] Negotiation successful
0929 20:29:03.154521 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":320,"threads_started":1}
W20260929 20:29:03.155676 31876 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44354
W20260929 20:29:03.156420 29650 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 31746: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:29:03.156580 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 31746
I20260929 20:29:03.162667 32049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:03.162064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:34227 (local address 127.0.0.1:58484)
0929 20:29:03.162321 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:03.162447 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:34227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 20:29:03.164091 32049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:03.163577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:34227 (local address 127.0.0.1:58496)
0929 20:29:03.163793 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:03.163923 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:34227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 20:29:03.169121 32049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:03.168560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:46299 (local address 127.0.0.1:40540)
0929 20:29:03.168786 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:03.168908 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.130:46299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 20:29:03.170413 32049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:03.169913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:46299 (local address 127.0.0.1:40556)
0929 20:29:03.170141 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:03.170252 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.130:46299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 20:29:03.184079 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 31363
I20260929 20:29:03.185542 31435 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:29:03.305822 32049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:03.304983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:46299 (local address 127.0.0.1:40566)
0929 20:29:03.305453 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:03.305565 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.130:46299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281}
I20260929 20:29:03.306185 32052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:03.305070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:34227 (local address 127.0.0.1:58498)
0929 20:29:03.305647 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:03.305933 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:34227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":308,"threads_started":1}
I20260929 20:29:03.306576 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 31363
2026-09-29T20:29:03Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8867 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:29:03 dist-test-slave-p29l krb5kdc[32055](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:29:03 dist-test-slave-p29l krb5kdc[32055](info): set up 2 sockets
Sep 29 20:29:03 dist-test-slave-p29l krb5kdc[32055](info): commencing operation
krb5kdc: starting...
W20260929 20:29:04.595971 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.241s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:29:04 dist-test-slave-p29l krb5kdc[32055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:29:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:29:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.244.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
I20260929 20:29:04.811745 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:44255
--webserver_interface=127.28.244.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:36547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:44255
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:29:05.170389 32068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:05.170711 32068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:05.170914 32068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:05.178522 32068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:29:05.178656 32068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:05.178746 32068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:05.178856 32068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:29:05.178915 32068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:29:05.189589 32068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:36547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:44255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:44255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:05.191064 32068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:05.192589 32068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:05.205864 32073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:05.205830 32074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:05.205894 32076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:05.206478 32068 server_base.cc:1034] running on GCE node
I20260929 20:29:05.207454 32068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:05.209110 32068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:05.210369 32068 hybrid_clock.cc:648] HybridClock initialized: now 1790713745210319 us; error 53 us; skew 500 ppm
I20260929 20:29:05.210762 32068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:05 dist-test-slave-p29l krb5kdc[32055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713745, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:05.214279 32068 init.cc:377] Logged in from keytab as kudu/127.28.244.190@KRBTEST.COM (short username kudu)
I20260929 20:29:05.216081 32068 webserver.cc:460] Webserver started at http://127.28.244.190:45111/ using document root <none> and password file <none>
I20260929 20:29:05.216853 32068 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:05.217005 32068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:05.217353 32068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:05.220715 32068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "028fd17500534dedbf127c676471f279"
format_stamp: "Formatted at 2026-09-29 20:29:05 on dist-test-slave-p29l"
server_key: "6351a42ee51d6182b030bcfb41ee7144"
server_key_iv: "0709d183383e6d91b984285de41be9ca"
server_key_version: "encryptionkey@0"
I20260929 20:29:05.221640 32068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "028fd17500534dedbf127c676471f279"
format_stamp: "Formatted at 2026-09-29 20:29:05 on dist-test-slave-p29l"
server_key: "6351a42ee51d6182b030bcfb41ee7144"
server_key_iv: "0709d183383e6d91b984285de41be9ca"
server_key_version: "encryptionkey@0"
I20260929 20:29:05.228041 32068 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260929 20:29:05.232323 32083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:05.233982 32068 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 20:29:05.234236 32068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "028fd17500534dedbf127c676471f279"
format_stamp: "Formatted at 2026-09-29 20:29:05 on dist-test-slave-p29l"
server_key: "6351a42ee51d6182b030bcfb41ee7144"
server_key_iv: "0709d183383e6d91b984285de41be9ca"
server_key_version: "encryptionkey@0"
I20260929 20:29:05.234505 32068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:05.256392 32068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:05.261318 32068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:05.261651 32068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:05.281497 32068 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:44255
I20260929 20:29:05.281550 32134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:44255 every 8 connection(s)
I20260929 20:29:05.283069 32068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:29:05.284406 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 32068
I20260929 20:29:05.284657 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:29:05.285463 29650 external_mini_cluster.cc:1428] Setting key 497b8e04cf374ba89a1a96d16bc45b6e
I20260929 20:29:05.288810 32135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 20:29:05 dist-test-slave-p29l krb5kdc[32055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:29:05.303480 32135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 028fd17500534dedbf127c676471f279: Bootstrap starting.
I20260929 20:29:05.308256 32135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 028fd17500534dedbf127c676471f279: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:05.309624 32135 log.cc:824] T 00000000000000000000000000000000 P 028fd17500534dedbf127c676471f279: Log is configured to *not* fsync() on all Append() calls
I20260929 20:29:05.313539 32138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:05.287732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41848 (local address 127.28.244.190:44255)
0929 20:29:05.288682 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:05.288699 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:05.288757 (+ 58us) server_negotiation.cc:407] Connection header received
0929 20:29:05.291257 (+ 2500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:05.291292 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:05.291904 (+ 612us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:05.292468 (+ 564us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:05.294193 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:05.296591 (+ 2398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:05.298141 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:05.299287 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:05.299543 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:05.303763 (+ 4220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:05.303775 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:05.303797 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:05.303880 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:05.308159 (+ 4279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:05.309031 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:05.309040 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:05.309049 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:05.309275 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:05.309754 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:05.309759 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:05.309762 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:05.310592 (+ 830us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:05.310849 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:05.311541 (+ 692us) server_negotiation.cc:299] Negotiation successful
0929 20:29:05.312054 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":343,"threads_started":1}
I20260929 20:29:05.313612 32135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 028fd17500534dedbf127c676471f279: No bootstrap required, opened a new log
I20260929 20:29:05.322049 32135 raft_consensus.cc:348] T 00000000000000000000000000000000 P 028fd17500534dedbf127c676471f279 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028fd17500534dedbf127c676471f279" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 44255 } }
I20260929 20:29:05.322520 32135 raft_consensus.cc:374] T 00000000000000000000000000000000 P 028fd17500534dedbf127c676471f279 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:05.322635 32135 raft_consensus.cc:729] T 00000000000000000000000000000000 P 028fd17500534dedbf127c676471f279 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 028fd17500534dedbf127c676471f279, State: Initialized, Role: FOLLOWER
I20260929 20:29:05.323361 32135 consensus_queue.cc:260] T 00000000000000000000000000000000 P 028fd17500534dedbf127c676471f279 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028fd17500534dedbf127c676471f279" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 44255 } }
I20260929 20:29:05.323673 32135 raft_consensus.cc:388] T 00000000000000000000000000000000 P 028fd17500534dedbf127c676471f279 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:29:05.323832 32135 raft_consensus.cc:482] T 00000000000000000000000000000000 P 028fd17500534dedbf127c676471f279 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:29:05.324033 32135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 028fd17500534dedbf127c676471f279 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:05.327512 32135 raft_consensus.cc:504] T 00000000000000000000000000000000 P 028fd17500534dedbf127c676471f279 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028fd17500534dedbf127c676471f279" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 44255 } }
I20260929 20:29:05.328228 32135 leader_election.cc:302] T 00000000000000000000000000000000 P 028fd17500534dedbf127c676471f279 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 028fd17500534dedbf127c676471f279; no voters:
I20260929 20:29:05.328926 32135 leader_election.cc:288] T 00000000000000000000000000000000 P 028fd17500534dedbf127c676471f279 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:29:05.329596 32140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 028fd17500534dedbf127c676471f279 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:29:05.330238 32140 raft_consensus.cc:686] T 00000000000000000000000000000000 P 028fd17500534dedbf127c676471f279 [term 1 LEADER]: Becoming Leader. State: Replica: 028fd17500534dedbf127c676471f279, State: Running, Role: LEADER
I20260929 20:29:05.330826 32140 consensus_queue.cc:237] T 00000000000000000000000000000000 P 028fd17500534dedbf127c676471f279 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028fd17500534dedbf127c676471f279" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 44255 } }
I20260929 20:29:05.331372 32135 sys_catalog.cc:564] T 00000000000000000000000000000000 P 028fd17500534dedbf127c676471f279 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:29:05.333977 32142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 028fd17500534dedbf127c676471f279 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 028fd17500534dedbf127c676471f279. Latest consensus state: current_term: 1 leader_uuid: "028fd17500534dedbf127c676471f279" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028fd17500534dedbf127c676471f279" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 44255 } } }
I20260929 20:29:05.334401 32142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 028fd17500534dedbf127c676471f279 [sys.catalog]: This master's current role is: LEADER
I20260929 20:29:05.334921 32141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 028fd17500534dedbf127c676471f279 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "028fd17500534dedbf127c676471f279" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028fd17500534dedbf127c676471f279" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 44255 } } }
I20260929 20:29:05.335242 32141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 028fd17500534dedbf127c676471f279 [sys.catalog]: This master's current role is: LEADER
I20260929 20:29:05.335640 32147 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:29:05.342628 32147 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:29:05.353506 32147 catalog_manager.cc:1347] Generated new cluster ID: df2904b0e2ae48e7b39a5371936f430e
I20260929 20:29:05.353645 32147 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:29:05.378579 32147 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:29:05.380158 32147 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:29:05.391412 32147 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 028fd17500534dedbf127c676471f279: Generated new TSK 0
I20260929 20:29:05.392704 32147 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.244.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab.
Entry for principal kudu/127.28.244.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab.
I20260929 20:29:05.451743 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.129:0
--local_ip_for_outbound_sockets=127.28.244.129
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:44255
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:36547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:29:05.802861 32161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:05.803171 32161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:05.803331 32161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:05.810961 32161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:05.811089 32161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:05.811298 32161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.129
I20260929 20:29:05.821913 32161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:36547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:44255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.129
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:05.823346 32161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:05.824846 32161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:05.840039 32166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:05.840129 32169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:05.840108 32167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:05.840984 32161 server_base.cc:1034] running on GCE node
I20260929 20:29:05.841629 32161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:05.842669 32161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:05.843967 32161 hybrid_clock.cc:648] HybridClock initialized: now 1790713745843896 us; error 56 us; skew 500 ppm
I20260929 20:29:05.844344 32161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:05 dist-test-slave-p29l krb5kdc[32055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713745, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:05.847885 32161 init.cc:377] Logged in from keytab as kudu/127.28.244.129@KRBTEST.COM (short username kudu)
I20260929 20:29:05.849550 32161 webserver.cc:460] Webserver started at http://127.28.244.129:46259/ using document root <none> and password file <none>
I20260929 20:29:05.850306 32161 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:05.850451 32161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:05.850805 32161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:05.854049 32161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-0/data/instance:
uuid: "d0b6fbbedf194a0fb0cd89e8f98d64e9"
format_stamp: "Formatted at 2026-09-29 20:29:05 on dist-test-slave-p29l"
server_key: "4c7a216c2d50cce3a3f991c837f2e09c"
server_key_iv: "3e26ff2ce376805df487f4afbf65c104"
server_key_version: "encryptionkey@0"
I20260929 20:29:05.854980 32161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance:
uuid: "d0b6fbbedf194a0fb0cd89e8f98d64e9"
format_stamp: "Formatted at 2026-09-29 20:29:05 on dist-test-slave-p29l"
server_key: "4c7a216c2d50cce3a3f991c837f2e09c"
server_key_iv: "3e26ff2ce376805df487f4afbf65c104"
server_key_version: "encryptionkey@0"
I20260929 20:29:05.861146 32161 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 20:29:05.864988 32176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:05.866535 32161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:29:05.866752 32161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-0/wal
uuid: "d0b6fbbedf194a0fb0cd89e8f98d64e9"
format_stamp: "Formatted at 2026-09-29 20:29:05 on dist-test-slave-p29l"
server_key: "4c7a216c2d50cce3a3f991c837f2e09c"
server_key_iv: "3e26ff2ce376805df487f4afbf65c104"
server_key_version: "encryptionkey@0"
I20260929 20:29:05.866987 32161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:05.883797 32161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:05.888674 32161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:05.889020 32161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:05.890208 32161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:29:05.891683 32161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:29:05.891793 32161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:05.891989 32161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:29:05.892066 32161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:05.926844 32161 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.129:37237
I20260929 20:29:05.926856 32288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.129:37237 every 8 connection(s)
I20260929 20:29:05.928350 32161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
I20260929 20:29:05.933779 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 32161
I20260929 20:29:05.934036 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance
I20260929 20:29:05.934654 29650 external_mini_cluster.cc:1428] Setting key 66500b46077ae6c989d3bbe21dd8cab6
Sep 29 20:29:05 dist-test-slave-p29l krb5kdc[32055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713745, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:29:05.950755 32291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:05.931094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:48859 (local address 127.28.244.190:44255)
0929 20:29:05.931603 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:05.931607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:05.932825 (+ 1218us) server_negotiation.cc:407] Connection header received
0929 20:29:05.935098 (+ 2273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:05.935104 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:05.935206 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:05.935331 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:05.937249 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:05.938098 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:05.939991 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:05.940424 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:05.940575 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:05.944421 (+ 3846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:05.944427 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:05.944430 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:05.944467 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:05.947307 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:05.948302 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:05.948307 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:05.948310 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:05.948446 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:05.948907 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:05.948913 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:05.948921 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:05.949198 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:05.949358 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:05.950132 (+ 774us) server_negotiation.cc:299] Negotiation successful
0929 20:29:05.950369 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":286,"threads_started":1}
I20260929 20:29:05.951871 32292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:05.931742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:44255 (local address 127.28.244.129:48859)
0929 20:29:05.932653 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:05.932697 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:05.934764 (+ 2067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:05.935622 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:05.935635 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:05.936438 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:05.937070 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:05.937089 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:05.938263 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:05.938275 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:05.939777 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:05.939792 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:05.940677 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:05.940685 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:05.940994 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:05.941760 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:05.941785 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:05.944116 (+ 2331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:05.947536 (+ 3420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:05.947554 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:05.947571 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:05.948151 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:05.948567 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:05.948572 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:05.948576 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:05.948780 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:05.949331 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:05.949340 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:05.949728 (+ 388us) client_negotiation.cc:769] Sending connection context
0929 20:29:05.950011 (+ 283us) client_negotiation.cc:240] Negotiation successful
0929 20:29:05.950601 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":375,"threads_started":1}
WARNING: no policy specified for kudu/127.28.244.130@KRBTEST.COM; defaulting to no policy
I20260929 20:29:05.953526 32289 heartbeater.cc:347] Connected to a master server at 127.28.244.190:44255
I20260929 20:29:05.953912 32289 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.130@KRBTEST.COM" created.
I20260929 20:29:05.954821 32289 heartbeater.cc:511] Master 127.28.244.190:44255 requested a full tablet report, sending...
I20260929 20:29:05.957172 32100 ts_manager.cc:194] Registered new tserver with Master: d0b6fbbedf194a0fb0cd89e8f98d64e9 (127.28.244.129:37237)
I20260929 20:29:05.958734 32100 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.129@KRBTEST.COM'} at 127.28.244.129:48859
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab.
Entry for principal kudu/127.28.244.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab.
I20260929 20:29:05.979620 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.130:0
--local_ip_for_outbound_sockets=127.28.244.130
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:44255
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:36547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:29:06.331496 32295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:06.331847 32295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:06.332010 32295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:06.339910 32295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:06.340040 32295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:06.340157 32295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.130
I20260929 20:29:06.350665 32295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:36547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:44255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.130
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:06.352098 32295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:06.353479 32295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:06.368901 32303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:06.368942 32301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:06.369809 32300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:06.373678 32295 server_base.cc:1034] running on GCE node
I20260929 20:29:06.374454 32295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:06.375512 32295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:06.376765 32295 hybrid_clock.cc:648] HybridClock initialized: now 1790713746376719 us; error 65 us; skew 500 ppm
I20260929 20:29:06.377159 32295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:06 dist-test-slave-p29l krb5kdc[32055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713746, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:06.380685 32295 init.cc:377] Logged in from keytab as kudu/127.28.244.130@KRBTEST.COM (short username kudu)
I20260929 20:29:06.382390 32295 webserver.cc:460] Webserver started at http://127.28.244.130:38057/ using document root <none> and password file <none>
I20260929 20:29:06.383181 32295 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:06.383291 32295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:06.383560 32295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:06.386967 32295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-1/data/instance:
uuid: "940fc84bacbf4b62b44c58fbad5eb7e7"
format_stamp: "Formatted at 2026-09-29 20:29:06 on dist-test-slave-p29l"
server_key: "9ab1b5038cfa959c6b06c705d70fc73a"
server_key_iv: "d5c09706f195c7dabcdb10ff22bd56b2"
server_key_version: "encryptionkey@0"
I20260929 20:29:06.387933 32295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance:
uuid: "940fc84bacbf4b62b44c58fbad5eb7e7"
format_stamp: "Formatted at 2026-09-29 20:29:06 on dist-test-slave-p29l"
server_key: "9ab1b5038cfa959c6b06c705d70fc73a"
server_key_iv: "d5c09706f195c7dabcdb10ff22bd56b2"
server_key_version: "encryptionkey@0"
I20260929 20:29:06.393690 32295 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 20:29:06.397671 32310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:06.399200 32295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:29:06.399359 32295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-1/wal
uuid: "940fc84bacbf4b62b44c58fbad5eb7e7"
format_stamp: "Formatted at 2026-09-29 20:29:06 on dist-test-slave-p29l"
server_key: "9ab1b5038cfa959c6b06c705d70fc73a"
server_key_iv: "d5c09706f195c7dabcdb10ff22bd56b2"
server_key_version: "encryptionkey@0"
I20260929 20:29:06.399562 32295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:06.416712 32295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:06.422287 32295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:06.422626 32295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:06.423672 32295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:29:06.425151 32295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:29:06.425263 32295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:06.425416 32295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:29:06.425493 32295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:06.460467 32295 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.130:34575
I20260929 20:29:06.460512 32422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.130:34575 every 8 connection(s)
I20260929 20:29:06.461901 32295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
I20260929 20:29:06.470697 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 32295
I20260929 20:29:06.470897 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance
I20260929 20:29:06.471397 29650 external_mini_cluster.cc:1428] Setting key b09b9f29a6d0bfb6412ced2ffd25ed10
Sep 29 20:29:06 dist-test-slave-p29l krb5kdc[32055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713746, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:29:06.484282 32425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:06.464672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:55583 (local address 127.28.244.190:44255)
0929 20:29:06.465122 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:06.465128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:06.466264 (+ 1136us) server_negotiation.cc:407] Connection header received
0929 20:29:06.468300 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:06.468304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:06.468386 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:06.468489 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:06.470255 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:06.471249 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:06.473137 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:06.473605 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:06.473756 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:06.477716 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:06.477723 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:06.477728 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:06.477767 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:06.480728 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:06.481649 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:06.481654 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:06.481657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:06.481800 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:06.482290 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:06.482301 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:06.482303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:06.482608 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:06.482775 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:06.483591 (+ 816us) server_negotiation.cc:299] Negotiation successful
0929 20:29:06.483894 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":204,"threads_started":1}
I20260929 20:29:06.485374 32426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:06.465307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:44255 (local address 127.28.244.130:55583)
0929 20:29:06.466078 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:06.466121 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:06.467995 (+ 1874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:06.468750 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:06.468759 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:06.469340 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:06.470046 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:06.470069 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:06.471445 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:06.471449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:06.472967 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:06.472982 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:06.473871 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:06.473880 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:06.474134 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:06.474946 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:06.474973 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:06.477407 (+ 2434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:06.480935 (+ 3528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:06.480948 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:06.480973 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:06.481489 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:06.481924 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:06.481929 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:06.481933 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:06.482151 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:06.482748 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:06.482758 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:06.483180 (+ 422us) client_negotiation.cc:769] Sending connection context
0929 20:29:06.483475 (+ 295us) client_negotiation.cc:240] Negotiation successful
0929 20:29:06.484140 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":304,"threads_started":1}
I20260929 20:29:06.487043 32423 heartbeater.cc:347] Connected to a master server at 127.28.244.190:44255
I20260929 20:29:06.487416 32423 heartbeater.cc:464] Registering TS with master...
I20260929 20:29:06.488477 32423 heartbeater.cc:511] Master 127.28.244.190:44255 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.244.131@KRBTEST.COM; defaulting to no policy
I20260929 20:29:06.490350 32100 ts_manager.cc:194] Registered new tserver with Master: 940fc84bacbf4b62b44c58fbad5eb7e7 (127.28.244.130:34575)
I20260929 20:29:06.491211 32100 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.130@KRBTEST.COM'} at 127.28.244.130:55583
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab.
Entry for principal kudu/127.28.244.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab.
I20260929 20:29:06.516049 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.131:0
--local_ip_for_outbound_sockets=127.28.244.131
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:44255
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:36547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:29:06.858100 32429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:06.858418 32429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:06.858584 32429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:06.866057 32429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:06.866195 32429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:06.866382 32429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.131
I20260929 20:29:06.876933 32429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:36547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:44255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.131
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:06.878357 32429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:06.879781 32429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:06.894016 32434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:06.893993 32435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:06.893929 32437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:06.894793 32429 server_base.cc:1034] running on GCE node
I20260929 20:29:06.895511 32429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:06.896648 32429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:06.897912 32429 hybrid_clock.cc:648] HybridClock initialized: now 1790713746897849 us; error 54 us; skew 500 ppm
I20260929 20:29:06.898309 32429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:06 dist-test-slave-p29l krb5kdc[32055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713746, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:06.901710 32429 init.cc:377] Logged in from keytab as kudu/127.28.244.131@KRBTEST.COM (short username kudu)
I20260929 20:29:06.903431 32429 webserver.cc:460] Webserver started at http://127.28.244.131:36405/ using document root <none> and password file <none>
I20260929 20:29:06.904321 32429 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:06.904435 32429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:06.904871 32429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:06.908206 32429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-2/data/instance:
uuid: "2b49efe5f1434009af525f2ebcad58e1"
format_stamp: "Formatted at 2026-09-29 20:29:06 on dist-test-slave-p29l"
server_key: "c0768d7a22e45040d57b455f11698db8"
server_key_iv: "abd487b9028f292aa03b2ca226e4b2b6"
server_key_version: "encryptionkey@0"
I20260929 20:29:06.909113 32429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance:
uuid: "2b49efe5f1434009af525f2ebcad58e1"
format_stamp: "Formatted at 2026-09-29 20:29:06 on dist-test-slave-p29l"
server_key: "c0768d7a22e45040d57b455f11698db8"
server_key_iv: "abd487b9028f292aa03b2ca226e4b2b6"
server_key_version: "encryptionkey@0"
I20260929 20:29:06.915086 32429 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260929 20:29:06.919860 32444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:06.921469 32429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:29:06.921808 32429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-2/wal
uuid: "2b49efe5f1434009af525f2ebcad58e1"
format_stamp: "Formatted at 2026-09-29 20:29:06 on dist-test-slave-p29l"
server_key: "c0768d7a22e45040d57b455f11698db8"
server_key_iv: "abd487b9028f292aa03b2ca226e4b2b6"
server_key_version: "encryptionkey@0"
I20260929 20:29:06.922058 32429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:06.946692 32429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:06.951722 32429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:06.952086 32429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:06.953150 32429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:29:06.954653 32429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:29:06.954731 32429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:06.954818 32429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:29:06.954860 32429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:06.962112 32289 heartbeater.cc:503] Master 127.28.244.190:44255 was elected leader, sending a full tablet report...
I20260929 20:29:06.990245 32429 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.131:33693
I20260929 20:29:06.990337 32556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.131:33693 every 8 connection(s)
I20260929 20:29:06.991677 32429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
I20260929 20:29:06.996627 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 32429
I20260929 20:29:06.996822 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance
I20260929 20:29:06.997253 29650 external_mini_cluster.cc:1428] Setting key ea5ca75008ce7a6aff516f753b43a792
Sep 29 20:29:06 dist-test-slave-p29l krb5kdc[32055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713746, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:29:07.015064 32559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:06.994410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:58817 (local address 127.28.244.190:44255)
0929 20:29:06.994909 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:06.994914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:06.995796 (+ 882us) server_negotiation.cc:407] Connection header received
0929 20:29:06.998037 (+ 2241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:06.998044 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:06.998140 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:06.998263 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:07.000607 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:07.001621 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:07.003470 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:07.004081 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:07.004196 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:07.008085 (+ 3889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:07.008091 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:07.008095 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:07.008133 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:07.011029 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:07.012104 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:07.012110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:07.012113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:07.012295 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:07.012930 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:07.012935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:07.012938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:07.013321 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:07.013483 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:07.014515 (+ 1032us) server_negotiation.cc:299] Negotiation successful
0929 20:29:07.014727 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":274,"threads_started":1}
I20260929 20:29:07.016060 32560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:06.994879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:44255 (local address 127.28.244.131:58817)
0929 20:29:06.995629 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:06.995669 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:06.997626 (+ 1957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:06.998665 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:06.998678 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:06.999401 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:07.000310 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:07.000346 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:07.001806 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:07.001812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:07.003232 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:07.003248 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:07.004312 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:07.004318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:07.004585 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:07.005353 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:07.005378 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:07.007738 (+ 2360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:07.011209 (+ 3471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:07.011218 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:07.011236 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:07.011965 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:07.012431 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:07.012435 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:07.012437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:07.012779 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:07.013483 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:07.013493 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:07.014059 (+ 566us) client_negotiation.cc:769] Sending connection context
0929 20:29:07.014321 (+ 262us) client_negotiation.cc:240] Negotiation successful
0929 20:29:07.014787 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":265,"threads_started":1}
I20260929 20:29:07.017740 32557 heartbeater.cc:347] Connected to a master server at 127.28.244.190:44255
I20260929 20:29:07.018149 32557 heartbeater.cc:464] Registering TS with master...
I20260929 20:29:07.018967 32557 heartbeater.cc:511] Master 127.28.244.190:44255 requested a full tablet report, sending...
I20260929 20:29:07.020782 32100 ts_manager.cc:194] Registered new tserver with Master: 2b49efe5f1434009af525f2ebcad58e1 (127.28.244.131:33693)
I20260929 20:29:07.021653 32100 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.131@KRBTEST.COM'} at 127.28.244.131:58817
I20260929 20:29:07.027148 29650 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 20:29:07.060801 32559 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:29:07.058191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41862 (local address 127.28.244.190:44255)
0929 20:29:07.058430 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:07.058436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:07.058748 (+ 312us) server_negotiation.cc:407] Connection header received
0929 20:29:07.058950 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:07.058955 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:07.059026 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:07.059123 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:07.060587 (+ 1464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41862: BlockingRecv error: recv got EOF from 127.0.0.1:41862 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:29:07.064179 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 32161
W20260929 20:29:07.067241 29650 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 32161: Not authorized: Client connection negotiation failed: client connection to 127.28.244.129:37237: server requires authentication, but client does not have Kerberos credentials available
I20260929 20:29:07.067458 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 32161
I20260929 20:29:07.084136 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 32295
W20260929 20:29:07.087136 29650 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 32295: Not authorized: Client connection negotiation failed: client connection to 127.28.244.130:34575: server requires authentication, but client does not have Kerberos credentials available
I20260929 20:29:07.087302 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 32295
I20260929 20:29:07.103772 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 32429
W20260929 20:29:07.107040 29650 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 32429: Not authorized: Client connection negotiation failed: client connection to 127.28.244.131:33693: server requires authentication, but client does not have Kerberos credentials available
I20260929 20:29:07.107165 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 32429
I20260929 20:29:07.123672 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 32068
I20260929 20:29:07.125244 32130 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:29:07.232348 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 32068
2026-09-29T20:29:07Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3920 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:29:07 dist-test-slave-p29l krb5kdc[32575](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:29:07 dist-test-slave-p29l krb5kdc[32575](info): set up 2 sockets
Sep 29 20:29:07 dist-test-slave-p29l krb5kdc[32575](info): commencing operation
krb5kdc: starting...
W20260929 20:29:08.335665 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:29:08 dist-test-slave-p29l krb5kdc[32575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:29:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:29:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.244.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
I20260929 20:29:08.541013 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:35481
--webserver_interface=127.28.244.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:35829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:35481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:29:08.887563 32588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:08.888038 32588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:08.888199 32588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:08.895948 32588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:29:08.896116 32588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:08.896224 32588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:08.896337 32588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:29:08.896426 32588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:29:08.906939 32588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:35829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:35481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:35481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:08.908428 32588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:08.909915 32588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:08.927251 32593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:08.927251 32596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:08.927554 32594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:08.928373 32588 server_base.cc:1034] running on GCE node
I20260929 20:29:08.929353 32588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:08.931027 32588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:08.932366 32588 hybrid_clock.cc:648] HybridClock initialized: now 1790713748932292 us; error 75 us; skew 500 ppm
I20260929 20:29:08.932850 32588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:08 dist-test-slave-p29l krb5kdc[32575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713748, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:08.942021 32588 init.cc:377] Logged in from keytab as kudu/127.28.244.190@KRBTEST.COM (short username kudu)
I20260929 20:29:08.943897 32588 webserver.cc:460] Webserver started at http://127.28.244.190:41277/ using document root <none> and password file <none>
I20260929 20:29:08.944715 32588 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:08.944876 32588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:08.945215 32588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:08.948549 32588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "f717e97f7cac4c43b2e45344a83a3b10"
format_stamp: "Formatted at 2026-09-29 20:29:08 on dist-test-slave-p29l"
server_key: "7cb004f56557e838a5ddc9a6949ec1c1"
server_key_iv: "4148a7a1fbb0554475b701370dcb47d9"
server_key_version: "encryptionkey@0"
I20260929 20:29:08.949421 32588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "f717e97f7cac4c43b2e45344a83a3b10"
format_stamp: "Formatted at 2026-09-29 20:29:08 on dist-test-slave-p29l"
server_key: "7cb004f56557e838a5ddc9a6949ec1c1"
server_key_iv: "4148a7a1fbb0554475b701370dcb47d9"
server_key_version: "encryptionkey@0"
I20260929 20:29:08.955065 32588 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:29:08.959291 32603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:08.960857 32588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260929 20:29:08.961082 32588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "f717e97f7cac4c43b2e45344a83a3b10"
format_stamp: "Formatted at 2026-09-29 20:29:08 on dist-test-slave-p29l"
server_key: "7cb004f56557e838a5ddc9a6949ec1c1"
server_key_iv: "4148a7a1fbb0554475b701370dcb47d9"
server_key_version: "encryptionkey@0"
I20260929 20:29:08.961321 32588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:08.978008 32588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:08.982758 32588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:08.983073 32588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:09.002907 32588 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:35481
I20260929 20:29:09.002959 32654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:35481 every 8 connection(s)
I20260929 20:29:09.004452 32588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:29:09.008522 32655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:09.011705 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 32588
I20260929 20:29:09.011952 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:29:09.012795 29650 external_mini_cluster.cc:1428] Setting key 569a2edf4f7dc2128ff7e38cbeb4ebeb
I20260929 20:29:09.021826 32655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f717e97f7cac4c43b2e45344a83a3b10: Bootstrap starting.
I20260929 20:29:09.026069 32655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f717e97f7cac4c43b2e45344a83a3b10: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:09.027557 32655 log.cc:824] T 00000000000000000000000000000000 P f717e97f7cac4c43b2e45344a83a3b10: Log is configured to *not* fsync() on all Append() calls
Sep 29 20:29:09 dist-test-slave-p29l krb5kdc[32575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:29:09.031039 32655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f717e97f7cac4c43b2e45344a83a3b10: No bootstrap required, opened a new log
I20260929 20:29:09.038240 32658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:09.014827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35442 (local address 127.28.244.190:35481)
0929 20:29:09.015648 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:09.015661 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:09.015705 (+ 44us) server_negotiation.cc:407] Connection header received
0929 20:29:09.017451 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:09.017476 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:09.017934 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:09.018293 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:09.020283 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:09.021664 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:09.023516 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:09.024326 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:09.024471 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:09.029665 (+ 5194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:09.029675 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:09.029692 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:09.029735 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:09.033387 (+ 3652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:09.034414 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:09.034425 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:09.034433 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:09.034657 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:09.035136 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:09.035140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:09.035142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:09.035881 (+ 739us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:09.036145 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:09.036672 (+ 527us) server_negotiation.cc:299] Negotiation successful
0929 20:29:09.037084 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":390,"threads_started":1}
I20260929 20:29:09.040567 32655 raft_consensus.cc:348] T 00000000000000000000000000000000 P f717e97f7cac4c43b2e45344a83a3b10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f717e97f7cac4c43b2e45344a83a3b10" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 35481 } }
I20260929 20:29:09.040992 32655 raft_consensus.cc:374] T 00000000000000000000000000000000 P f717e97f7cac4c43b2e45344a83a3b10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:09.041113 32655 raft_consensus.cc:729] T 00000000000000000000000000000000 P f717e97f7cac4c43b2e45344a83a3b10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f717e97f7cac4c43b2e45344a83a3b10, State: Initialized, Role: FOLLOWER
I20260929 20:29:09.041793 32655 consensus_queue.cc:260] T 00000000000000000000000000000000 P f717e97f7cac4c43b2e45344a83a3b10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f717e97f7cac4c43b2e45344a83a3b10" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 35481 } }
I20260929 20:29:09.042032 32655 raft_consensus.cc:388] T 00000000000000000000000000000000 P f717e97f7cac4c43b2e45344a83a3b10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:29:09.042179 32655 raft_consensus.cc:482] T 00000000000000000000000000000000 P f717e97f7cac4c43b2e45344a83a3b10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:29:09.042354 32655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f717e97f7cac4c43b2e45344a83a3b10 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:09.042418 29650 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:29:09.044689 32655 raft_consensus.cc:504] T 00000000000000000000000000000000 P f717e97f7cac4c43b2e45344a83a3b10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f717e97f7cac4c43b2e45344a83a3b10" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 35481 } }
I20260929 20:29:09.045231 32655 leader_election.cc:302] T 00000000000000000000000000000000 P f717e97f7cac4c43b2e45344a83a3b10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f717e97f7cac4c43b2e45344a83a3b10; no voters:
I20260929 20:29:09.045843 32655 leader_election.cc:288] T 00000000000000000000000000000000 P f717e97f7cac4c43b2e45344a83a3b10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:29:09.046005 32660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f717e97f7cac4c43b2e45344a83a3b10 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:29:09.047219 32660 raft_consensus.cc:686] T 00000000000000000000000000000000 P f717e97f7cac4c43b2e45344a83a3b10 [term 1 LEADER]: Becoming Leader. State: Replica: f717e97f7cac4c43b2e45344a83a3b10, State: Running, Role: LEADER
I20260929 20:29:09.047825 32660 consensus_queue.cc:237] T 00000000000000000000000000000000 P f717e97f7cac4c43b2e45344a83a3b10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f717e97f7cac4c43b2e45344a83a3b10" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 35481 } }
I20260929 20:29:09.048221 32655 sys_catalog.cc:564] T 00000000000000000000000000000000 P f717e97f7cac4c43b2e45344a83a3b10 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:29:09.054759 32661 sys_catalog.cc:455] T 00000000000000000000000000000000 P f717e97f7cac4c43b2e45344a83a3b10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f717e97f7cac4c43b2e45344a83a3b10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f717e97f7cac4c43b2e45344a83a3b10" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 35481 } } }
I20260929 20:29:09.055169 32661 sys_catalog.cc:458] T 00000000000000000000000000000000 P f717e97f7cac4c43b2e45344a83a3b10 [sys.catalog]: This master's current role is: LEADER
I20260929 20:29:09.055790 32662 sys_catalog.cc:455] T 00000000000000000000000000000000 P f717e97f7cac4c43b2e45344a83a3b10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f717e97f7cac4c43b2e45344a83a3b10. Latest consensus state: current_term: 1 leader_uuid: "f717e97f7cac4c43b2e45344a83a3b10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f717e97f7cac4c43b2e45344a83a3b10" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 35481 } } }
I20260929 20:29:09.056087 32662 sys_catalog.cc:458] T 00000000000000000000000000000000 P f717e97f7cac4c43b2e45344a83a3b10 [sys.catalog]: This master's current role is: LEADER
I20260929 20:29:09.064459 32676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:29:09.070366 32676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:29:09.081590 32676 catalog_manager.cc:1347] Generated new cluster ID: 5e73587ed9a742e5b498665b88d2fb79
I20260929 20:29:09.081717 32676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:29:09.091966 32676 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:29:09.093986 32676 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:29:09.094911 32658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:09.084322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35448 (local address 127.28.244.190:35481)
0929 20:29:09.084512 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:09.084518 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:09.085235 (+ 717us) server_negotiation.cc:407] Connection header received
0929 20:29:09.085496 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:09.085502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:09.085600 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:09.085671 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:09.087258 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:09.088646 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:09.090582 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:09.091301 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:09.091447 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:09.092853 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:09.092862 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:09.092868 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:29:09.093803 (+ 935us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:09.093891 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:29:09.093931 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:09.094071 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:09.094344 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 20:29:09.094487 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":67}
I20260929 20:29:09.102099 32676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f717e97f7cac4c43b2e45344a83a3b10: Generated new TSK 0
I20260929 20:29:09.103121 32676 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:29:09.111639 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 32588
I20260929 20:29:09.113129 32650 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:29:09.205034 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 32588
2026-09-29T20:29:09Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1967 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:29:09 dist-test-slave-p29l krb5kdc[32689](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:29:09 dist-test-slave-p29l krb5kdc[32689](info): set up 2 sockets
Sep 29 20:29:09 dist-test-slave-p29l krb5kdc[32689](info): commencing operation
krb5kdc: starting...
W20260929 20:29:10.295760 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:29:10 dist-test-slave-p29l krb5kdc[32689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:29:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:29:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.244.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
I20260929 20:29:10.499264 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:38047
--webserver_interface=127.28.244.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:34933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:38047
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:29:10.834568 32702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:10.834887 32702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:10.835052 32702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:10.842417 32702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:29:10.842540 32702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:10.842634 32702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:10.842730 32702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:29:10.842849 32702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:29:10.853134 32702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:34933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:38047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:38047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:10.854636 32702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:10.856091 32702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:10.869410 32707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:10.869397 32708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:10.869401 32710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:10.870205 32702 server_base.cc:1034] running on GCE node
I20260929 20:29:10.871042 32702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:10.872680 32702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:10.873915 32702 hybrid_clock.cc:648] HybridClock initialized: now 1790713750873863 us; error 40 us; skew 500 ppm
I20260929 20:29:10.874336 32702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:10 dist-test-slave-p29l krb5kdc[32689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713750, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:10.878026 32702 init.cc:377] Logged in from keytab as kudu/127.28.244.190@KRBTEST.COM (short username kudu)
I20260929 20:29:10.879905 32702 webserver.cc:460] Webserver started at http://127.28.244.190:45383/ using document root <none> and password file <none>
I20260929 20:29:10.880738 32702 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:10.880905 32702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:10.881246 32702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:10.884804 32702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "2fbb66f6f0cd460581a0170c39716146"
format_stamp: "Formatted at 2026-09-29 20:29:10 on dist-test-slave-p29l"
server_key: "81e217a8d5c1920ede0503a345523e68"
server_key_iv: "b1c9d5cd0e66ddc360f5ab9767978ea6"
server_key_version: "encryptionkey@0"
I20260929 20:29:10.885685 32702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "2fbb66f6f0cd460581a0170c39716146"
format_stamp: "Formatted at 2026-09-29 20:29:10 on dist-test-slave-p29l"
server_key: "81e217a8d5c1920ede0503a345523e68"
server_key_iv: "b1c9d5cd0e66ddc360f5ab9767978ea6"
server_key_version: "encryptionkey@0"
I20260929 20:29:10.891745 32702 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 20:29:10.896085 32717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:10.897710 32702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:29:10.897928 32702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "2fbb66f6f0cd460581a0170c39716146"
format_stamp: "Formatted at 2026-09-29 20:29:10 on dist-test-slave-p29l"
server_key: "81e217a8d5c1920ede0503a345523e68"
server_key_iv: "b1c9d5cd0e66ddc360f5ab9767978ea6"
server_key_version: "encryptionkey@0"
I20260929 20:29:10.898149 32702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:10.918612 32702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:10.923336 32702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:10.923648 32702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:10.942821 300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:38047 every 8 connection(s)
I20260929 20:29:10.942818 32702 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:38047
I20260929 20:29:10.944397 32702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:29:10.948890 301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:10.949259 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 32702
I20260929 20:29:10.949466 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:29:10.950145 29650 external_mini_cluster.cc:1428] Setting key abc83d82ffebb824f42f29896f781442
I20260929 20:29:10.961869 301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fbb66f6f0cd460581a0170c39716146: Bootstrap starting.
Sep 29 20:29:10 dist-test-slave-p29l krb5kdc[32689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:29:10.965427 301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fbb66f6f0cd460581a0170c39716146: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:10.967000 301 log.cc:824] T 00000000000000000000000000000000 P 2fbb66f6f0cd460581a0170c39716146: Log is configured to *not* fsync() on all Append() calls
I20260929 20:29:10.970921 301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fbb66f6f0cd460581a0170c39716146: No bootstrap required, opened a new log
I20260929 20:29:10.975045 304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:10.952269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50490 (local address 127.28.244.190:38047)
0929 20:29:10.953122 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:10.953136 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:10.953180 (+ 44us) server_negotiation.cc:407] Connection header received
0929 20:29:10.955277 (+ 2097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:10.955304 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:10.955777 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:10.956287 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:10.957778 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:10.959162 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:10.960531 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:10.961249 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:10.961476 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:10.965823 (+ 4347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:10.965836 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:10.965865 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:10.965934 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:10.970015 (+ 4081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:10.970808 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:10.970817 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:10.970828 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:10.971092 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:10.971468 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:10.971474 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:10.971479 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:10.972267 (+ 788us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:10.972635 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:10.973016 (+ 381us) server_negotiation.cc:299] Negotiation successful
0929 20:29:10.973540 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":347,"threads_started":1}
I20260929 20:29:10.980614 301 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fbb66f6f0cd460581a0170c39716146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fbb66f6f0cd460581a0170c39716146" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 38047 } }
I20260929 20:29:10.981068 301 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fbb66f6f0cd460581a0170c39716146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:10.981163 301 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fbb66f6f0cd460581a0170c39716146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fbb66f6f0cd460581a0170c39716146, State: Initialized, Role: FOLLOWER
I20260929 20:29:10.981801 301 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fbb66f6f0cd460581a0170c39716146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fbb66f6f0cd460581a0170c39716146" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 38047 } }
I20260929 20:29:10.982053 301 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fbb66f6f0cd460581a0170c39716146 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:29:10.982205 301 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fbb66f6f0cd460581a0170c39716146 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:29:10.982372 301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fbb66f6f0cd460581a0170c39716146 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:10.984629 301 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fbb66f6f0cd460581a0170c39716146 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fbb66f6f0cd460581a0170c39716146" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 38047 } }
I20260929 20:29:10.985244 301 leader_election.cc:302] T 00000000000000000000000000000000 P 2fbb66f6f0cd460581a0170c39716146 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fbb66f6f0cd460581a0170c39716146; no voters:
I20260929 20:29:10.985954 301 leader_election.cc:288] T 00000000000000000000000000000000 P 2fbb66f6f0cd460581a0170c39716146 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:29:10.986115 306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fbb66f6f0cd460581a0170c39716146 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:29:10.986797 306 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fbb66f6f0cd460581a0170c39716146 [term 1 LEADER]: Becoming Leader. State: Replica: 2fbb66f6f0cd460581a0170c39716146, State: Running, Role: LEADER
I20260929 20:29:10.987331 306 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fbb66f6f0cd460581a0170c39716146 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fbb66f6f0cd460581a0170c39716146" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 38047 } }
I20260929 20:29:10.988207 301 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fbb66f6f0cd460581a0170c39716146 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:29:10.989992 307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fbb66f6f0cd460581a0170c39716146 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fbb66f6f0cd460581a0170c39716146" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fbb66f6f0cd460581a0170c39716146" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 38047 } } }
I20260929 20:29:10.990537 308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fbb66f6f0cd460581a0170c39716146 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fbb66f6f0cd460581a0170c39716146. Latest consensus state: current_term: 1 leader_uuid: "2fbb66f6f0cd460581a0170c39716146" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fbb66f6f0cd460581a0170c39716146" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 38047 } } }
I20260929 20:29:10.990861 307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fbb66f6f0cd460581a0170c39716146 [sys.catalog]: This master's current role is: LEADER
I20260929 20:29:10.990885 308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fbb66f6f0cd460581a0170c39716146 [sys.catalog]: This master's current role is: LEADER
I20260929 20:29:10.993108 311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:29:11.000008 311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:29:11.009887 311 catalog_manager.cc:1347] Generated new cluster ID: c19bc4eb5e224cb3804b5cd1c3fe07bf
I20260929 20:29:11.010033 311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:29:11.033725 311 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:29:11.035609 311 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:29:11.048703 311 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2fbb66f6f0cd460581a0170c39716146: Generated new TSK 0
I20260929 20:29:11.049655 311 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.244.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab.
Entry for principal kudu/127.28.244.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab.
I20260929 20:29:11.108287 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.129:0
--local_ip_for_outbound_sockets=127.28.244.129
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:38047
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:34933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:29:11.459740 327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:11.460137 327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:11.460327 327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:11.468020 327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:11.468160 327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:11.468328 327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.129
I20260929 20:29:11.478649 327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:34933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:38047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.129
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:11.480157 327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:11.481545 327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:11.495954 332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:11.496052 335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:11.495935 333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:11.496706 327 server_base.cc:1034] running on GCE node
I20260929 20:29:11.497360 327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:11.498339 327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:11.499629 327 hybrid_clock.cc:648] HybridClock initialized: now 1790713751499576 us; error 62 us; skew 500 ppm
I20260929 20:29:11.499981 327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:11 dist-test-slave-p29l krb5kdc[32689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713751, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:11.503317 327 init.cc:377] Logged in from keytab as kudu/127.28.244.129@KRBTEST.COM (short username kudu)
I20260929 20:29:11.504892 327 webserver.cc:460] Webserver started at http://127.28.244.129:46041/ using document root <none> and password file <none>
I20260929 20:29:11.505630 327 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:11.505757 327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:11.506076 327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:11.509208 327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-0/data/instance:
uuid: "475e7d50e489428a87edc77bec1e7f41"
format_stamp: "Formatted at 2026-09-29 20:29:11 on dist-test-slave-p29l"
server_key: "b08e3996c5fca1e782579dccc50e83af"
server_key_iv: "f52baf42bd6621da6e7ad0d715cf279f"
server_key_version: "encryptionkey@0"
I20260929 20:29:11.510033 327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance:
uuid: "475e7d50e489428a87edc77bec1e7f41"
format_stamp: "Formatted at 2026-09-29 20:29:11 on dist-test-slave-p29l"
server_key: "b08e3996c5fca1e782579dccc50e83af"
server_key_iv: "f52baf42bd6621da6e7ad0d715cf279f"
server_key_version: "encryptionkey@0"
I20260929 20:29:11.515547 327 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260929 20:29:11.519317 342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:11.520898 327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:29:11.521106 327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-0/wal
uuid: "475e7d50e489428a87edc77bec1e7f41"
format_stamp: "Formatted at 2026-09-29 20:29:11 on dist-test-slave-p29l"
server_key: "b08e3996c5fca1e782579dccc50e83af"
server_key_iv: "f52baf42bd6621da6e7ad0d715cf279f"
server_key_version: "encryptionkey@0"
I20260929 20:29:11.521320 327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:11.544596 327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:11.549248 327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:11.549561 327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:11.550568 327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:29:11.552037 327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:29:11.552143 327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:11.552295 327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:29:11.552364 327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:11.586270 327 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.129:38835
I20260929 20:29:11.586331 454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.129:38835 every 8 connection(s)
I20260929 20:29:11.587687 327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
I20260929 20:29:11.589068 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 327
I20260929 20:29:11.589294 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance
I20260929 20:29:11.590180 29650 external_mini_cluster.cc:1428] Setting key 9aa413bcefd68bcda87db7e6ef24a985
Sep 29 20:29:11 dist-test-slave-p29l krb5kdc[32689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713751, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.244.130@KRBTEST.COM; defaulting to no policy
I20260929 20:29:11.610200 457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:11.591027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:53713 (local address 127.28.244.190:38047)
0929 20:29:11.591521 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:11.591525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:11.592818 (+ 1293us) server_negotiation.cc:407] Connection header received
0929 20:29:11.594850 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:11.594856 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:11.594949 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:11.595078 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:11.596808 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:11.597564 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:11.599323 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:11.599732 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:11.599900 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:11.603986 (+ 4086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:11.603992 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:11.603996 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:11.604033 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:11.606703 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:11.607585 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:11.607590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:11.607593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:11.607721 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:11.608234 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:11.608238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:11.608240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:11.608527 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:11.608689 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:11.609554 (+ 865us) server_negotiation.cc:299] Negotiation successful
0929 20:29:11.609821 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":263,"threads_started":1}
I20260929 20:29:11.611339 458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:11.591694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:38047 (local address 127.28.244.129:53713)
0929 20:29:11.592671 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:11.592712 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:11.594561 (+ 1849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:11.595351 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:11.595361 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:11.595989 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:11.596627 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:11.596646 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:11.597724 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:11.597730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:11.599166 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:11.599178 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:11.599994 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:11.600004 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:11.600263 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:11.601048 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:11.601072 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:11.603688 (+ 2616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:11.606887 (+ 3199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:11.606898 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:11.606925 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:11.607423 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:11.607891 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:11.607895 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:11.607898 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:11.608105 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:11.608661 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:11.608671 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:11.609148 (+ 477us) client_negotiation.cc:769] Sending connection context
0929 20:29:11.609427 (+ 279us) client_negotiation.cc:240] Negotiation successful
0929 20:29:11.610087 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":336,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.130@KRBTEST.COM" created.
I20260929 20:29:11.613204 455 heartbeater.cc:347] Connected to a master server at 127.28.244.190:38047
I20260929 20:29:11.613580 455 heartbeater.cc:464] Registering TS with master...
I20260929 20:29:11.614580 455 heartbeater.cc:511] Master 127.28.244.190:38047 requested a full tablet report, sending...
I20260929 20:29:11.617461 32734 ts_manager.cc:194] Registered new tserver with Master: 475e7d50e489428a87edc77bec1e7f41 (127.28.244.129:38835)
I20260929 20:29:11.619122 32734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.129@KRBTEST.COM'} at 127.28.244.129:53713
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab.
Entry for principal kudu/127.28.244.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab.
I20260929 20:29:11.638775 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.130:0
--local_ip_for_outbound_sockets=127.28.244.130
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:38047
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:34933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:29:11.984467 461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:11.984823 461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:11.984977 461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:11.992511 461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:11.992627 461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:11.992740 461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.130
I20260929 20:29:12.003150 461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:34933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:38047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.130
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:12.004526 461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:12.005782 461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:12.019022 467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:12.019181 466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:12.019150 469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:12.020133 461 server_base.cc:1034] running on GCE node
I20260929 20:29:12.020866 461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:12.021963 461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:12.023206 461 hybrid_clock.cc:648] HybridClock initialized: now 1790713752023139 us; error 58 us; skew 500 ppm
I20260929 20:29:12.023598 461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:12 dist-test-slave-p29l krb5kdc[32689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713752, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:12.027124 461 init.cc:377] Logged in from keytab as kudu/127.28.244.130@KRBTEST.COM (short username kudu)
I20260929 20:29:12.028827 461 webserver.cc:460] Webserver started at http://127.28.244.130:44165/ using document root <none> and password file <none>
I20260929 20:29:12.029623 461 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:12.029783 461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:12.030120 461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:12.033479 461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-1/data/instance:
uuid: "00b399030db24fc8bdf6df5de0c51b1a"
format_stamp: "Formatted at 2026-09-29 20:29:12 on dist-test-slave-p29l"
server_key: "6b45fcce4a9280099c702efe9057a500"
server_key_iv: "028dc91a85f058ef122317ab7e3fd50b"
server_key_version: "encryptionkey@0"
I20260929 20:29:12.034359 461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance:
uuid: "00b399030db24fc8bdf6df5de0c51b1a"
format_stamp: "Formatted at 2026-09-29 20:29:12 on dist-test-slave-p29l"
server_key: "6b45fcce4a9280099c702efe9057a500"
server_key_iv: "028dc91a85f058ef122317ab7e3fd50b"
server_key_version: "encryptionkey@0"
I20260929 20:29:12.040189 461 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 20:29:12.044118 476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:12.045562 461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:29:12.045750 461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-1/wal
uuid: "00b399030db24fc8bdf6df5de0c51b1a"
format_stamp: "Formatted at 2026-09-29 20:29:12 on dist-test-slave-p29l"
server_key: "6b45fcce4a9280099c702efe9057a500"
server_key_iv: "028dc91a85f058ef122317ab7e3fd50b"
server_key_version: "encryptionkey@0"
I20260929 20:29:12.045989 461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:12.075244 461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:12.079967 461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:12.080286 461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:12.081275 461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:29:12.082640 461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:29:12.082711 461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:12.082883 461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:29:12.082934 461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:12.116179 461 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.130:45745
I20260929 20:29:12.116222 588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.130:45745 every 8 connection(s)
I20260929 20:29:12.117569 461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
I20260929 20:29:12.119392 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 461
I20260929 20:29:12.119563 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance
I20260929 20:29:12.120097 29650 external_mini_cluster.cc:1428] Setting key 416fd6e460b8aa23b65a04d4ba7d8f2a
Sep 29 20:29:12 dist-test-slave-p29l krb5kdc[32689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713752, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.244.131@KRBTEST.COM; defaulting to no policy
I20260929 20:29:12.138202 591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:12.120343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:36427 (local address 127.28.244.190:38047)
0929 20:29:12.120888 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:12.120897 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:12.123885 (+ 2988us) server_negotiation.cc:407] Connection header received
0929 20:29:12.124054 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:12.124059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:12.124144 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:12.124267 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:12.125886 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:12.126709 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:12.128334 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:12.128743 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:12.128871 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:12.132321 (+ 3450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:12.132326 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:12.132329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:12.132362 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:12.134936 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:12.135692 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:12.135697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:12.135699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:12.135869 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:12.136297 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:12.136302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:12.136306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:12.136580 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:12.136747 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:12.137487 (+ 740us) server_negotiation.cc:299] Negotiation successful
0929 20:29:12.137715 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":272,"threads_started":1}
I20260929 20:29:12.139284 592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:12.121088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:38047 (local address 127.28.244.130:36427)
0929 20:29:12.121950 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:12.122006 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:12.123719 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:12.124525 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:12.124534 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:12.125048 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:12.125713 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:12.125731 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:12.126864 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:12.126868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:12.128175 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:12.128189 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:12.128974 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:12.128981 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:12.129214 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:12.129910 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:12.129933 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:12.132094 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:12.135088 (+ 2994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:12.135098 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:12.135118 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:12.135562 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:12.135982 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:12.135987 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:12.135991 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:12.136179 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:12.136711 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:12.136720 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:12.137119 (+ 399us) client_negotiation.cc:769] Sending connection context
0929 20:29:12.137384 (+ 265us) client_negotiation.cc:240] Negotiation successful
0929 20:29:12.137964 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":349,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.131@KRBTEST.COM" created.
I20260929 20:29:12.140776 589 heartbeater.cc:347] Connected to a master server at 127.28.244.190:38047
I20260929 20:29:12.141108 589 heartbeater.cc:464] Registering TS with master...
I20260929 20:29:12.141871 589 heartbeater.cc:511] Master 127.28.244.190:38047 requested a full tablet report, sending...
I20260929 20:29:12.143447 32734 ts_manager.cc:194] Registered new tserver with Master: 00b399030db24fc8bdf6df5de0c51b1a (127.28.244.130:45745)
I20260929 20:29:12.144361 32734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.130@KRBTEST.COM'} at 127.28.244.130:36427
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab.
Entry for principal kudu/127.28.244.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab.
I20260929 20:29:12.168742 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.131:0
--local_ip_for_outbound_sockets=127.28.244.131
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:38047
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:34933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:29:12.514760 595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:12.515117 595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:12.515277 595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:12.522804 595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:12.522979 595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:12.523149 595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.131
I20260929 20:29:12.533545 595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:34933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:38047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.131
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:12.534950 595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:12.536362 595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:12.549950 600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:12.550065 601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:12.550153 603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:12.551337 595 server_base.cc:1034] running on GCE node
I20260929 20:29:12.552085 595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:12.553102 595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:12.554340 595 hybrid_clock.cc:648] HybridClock initialized: now 1790713752554270 us; error 41 us; skew 500 ppm
I20260929 20:29:12.554726 595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:12 dist-test-slave-p29l krb5kdc[32689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713752, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:12.558156 595 init.cc:377] Logged in from keytab as kudu/127.28.244.131@KRBTEST.COM (short username kudu)
I20260929 20:29:12.559738 595 webserver.cc:460] Webserver started at http://127.28.244.131:45393/ using document root <none> and password file <none>
I20260929 20:29:12.560506 595 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:12.560642 595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:12.560964 595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:12.564145 595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-2/data/instance:
uuid: "5b7c2f5056f246688c9f80eede361b69"
format_stamp: "Formatted at 2026-09-29 20:29:12 on dist-test-slave-p29l"
server_key: "8649a8292a98d101fa88ed5d0a1fec46"
server_key_iv: "1c9d6339668d7c2ea2a5ae788c6c9d9f"
server_key_version: "encryptionkey@0"
I20260929 20:29:12.564997 595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance:
uuid: "5b7c2f5056f246688c9f80eede361b69"
format_stamp: "Formatted at 2026-09-29 20:29:12 on dist-test-slave-p29l"
server_key: "8649a8292a98d101fa88ed5d0a1fec46"
server_key_iv: "1c9d6339668d7c2ea2a5ae788c6c9d9f"
server_key_version: "encryptionkey@0"
I20260929 20:29:12.570446 595 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:29:12.573933 610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:12.575286 595 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:29:12.575484 595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-2/wal
uuid: "5b7c2f5056f246688c9f80eede361b69"
format_stamp: "Formatted at 2026-09-29 20:29:12 on dist-test-slave-p29l"
server_key: "8649a8292a98d101fa88ed5d0a1fec46"
server_key_iv: "1c9d6339668d7c2ea2a5ae788c6c9d9f"
server_key_version: "encryptionkey@0"
I20260929 20:29:12.575686 595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:12.604838 595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:12.609549 595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:12.609866 595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:12.610846 595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:29:12.612267 595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:29:12.612389 595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:12.612534 595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:29:12.612604 595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:12.622809 455 heartbeater.cc:503] Master 127.28.244.190:38047 was elected leader, sending a full tablet report...
I20260929 20:29:12.646961 595 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.131:35797
I20260929 20:29:12.646996 722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.131:35797 every 8 connection(s)
I20260929 20:29:12.648461 595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
I20260929 20:29:12.650509 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 595
I20260929 20:29:12.650696 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance
I20260929 20:29:12.651149 29650 external_mini_cluster.cc:1428] Setting key ac63820300b2fb2bd0a2c7772035c66c
Sep 29 20:29:12 dist-test-slave-p29l krb5kdc[32689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713752, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:29:12.669456 726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:12.652220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:53637 (local address 127.28.244.190:38047)
0929 20:29:12.652716 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:12.652724 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:12.653063 (+ 339us) server_negotiation.cc:407] Connection header received
0929 20:29:12.655260 (+ 2197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:12.655264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:12.655330 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:12.655422 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:12.657005 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:12.657771 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:12.659294 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:12.659683 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:12.659775 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:12.663179 (+ 3404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:12.663184 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:12.663187 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:12.663221 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:12.666195 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:12.667092 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:12.667097 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:12.667101 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:12.667249 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:12.667705 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:12.667708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:12.667710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:12.668070 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:12.668199 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:12.669010 (+ 811us) server_negotiation.cc:299] Negotiation successful
0929 20:29:12.669184 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":258,"threads_started":1}
I20260929 20:29:12.670380 725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:12.652032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:38047 (local address 127.28.244.131:53637)
0929 20:29:12.652885 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:12.652941 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:12.655005 (+ 2064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:12.655636 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:12.655645 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:12.656243 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:12.656829 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:12.656847 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:12.657922 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:12.657926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:12.659169 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:12.659181 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:12.659936 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:12.659942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:12.660134 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:12.660817 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:12.660840 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:12.662975 (+ 2135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:12.666326 (+ 3351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:12.666334 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:12.666394 (+ 60us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:12.666961 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:12.667386 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:12.667390 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:12.667392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:12.667586 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:12.668199 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:12.668205 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:12.668572 (+ 367us) client_negotiation.cc:769] Sending connection context
0929 20:29:12.668841 (+ 269us) client_negotiation.cc:240] Negotiation successful
0929 20:29:12.669253 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":378,"threads_started":1}
I20260929 20:29:12.671770 723 heartbeater.cc:347] Connected to a master server at 127.28.244.190:38047
I20260929 20:29:12.672088 723 heartbeater.cc:464] Registering TS with master...
I20260929 20:29:12.672881 723 heartbeater.cc:511] Master 127.28.244.190:38047 requested a full tablet report, sending...
I20260929 20:29:12.674256 32734 ts_manager.cc:194] Registered new tserver with Master: 5b7c2f5056f246688c9f80eede361b69 (127.28.244.131:35797)
I20260929 20:29:12.675048 32734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.131@KRBTEST.COM'} at 127.28.244.131:53637
I20260929 20:29:12.679575 29650 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 20:29:12 dist-test-slave-p29l krb5kdc[32689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713752, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 20:29:12 dist-test-slave-p29l krb5kdc[32689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713752, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:29:12.728946 726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:12.714923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50504 (local address 127.28.244.190:38047)
0929 20:29:12.715095 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:12.715100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:12.715546 (+ 446us) server_negotiation.cc:407] Connection header received
0929 20:29:12.715701 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:12.715705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:12.715790 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:12.715938 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:12.717291 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:12.718035 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:12.719238 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:12.719660 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:12.719757 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:12.723653 (+ 3896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:12.723658 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:12.723662 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:12.723698 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:12.726391 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:12.727038 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:12.727042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:12.727044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:12.727167 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:12.727479 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:12.727481 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:12.727483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:12.727892 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:12.728148 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:12.728420 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 20:29:12.728612 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 20:29:12.729499 32734 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50504
I20260929 20:29:12.731279 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 327
Sep 29 20:29:12 dist-test-slave-p29l krb5kdc[32689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713752, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.244.129@KRBTEST.COM
I20260929 20:29:12.746950 735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:12.732386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35126 (local address 127.28.244.129:38835)
0929 20:29:12.732962 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:12.732969 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:12.732994 (+ 25us) server_negotiation.cc:407] Connection header received
0929 20:29:12.733054 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:12.733060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:12.733264 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:12.733395 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:12.734531 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:12.735610 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:12.736833 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:12.737424 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:12.737540 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:12.741387 (+ 3847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:12.741399 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:12.741414 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:12.741474 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:12.744400 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:12.745061 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:12.745069 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:12.745071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:12.745204 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:12.745518 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:12.745521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:12.745523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:12.745879 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:12.746117 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:12.746323 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 20:29:12.746535 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":308,"threads_started":1}
W20260929 20:29:12.748312 450 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35126
W20260929 20:29:12.749457 29650 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 327: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:29:12.749671 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 327
I20260929 20:29:12.765764 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 461
Sep 29 20:29:12 dist-test-slave-p29l krb5kdc[32689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713752, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.244.130@KRBTEST.COM
I20260929 20:29:12.782361 736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:12.766705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47514 (local address 127.28.244.130:45745)
0929 20:29:12.767222 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:12.767229 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:12.767254 (+ 25us) server_negotiation.cc:407] Connection header received
0929 20:29:12.767341 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:12.767347 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:12.767559 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:12.767733 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:12.769011 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:12.770098 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:12.771539 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:12.772263 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:12.772442 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:12.776680 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:12.776693 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:12.776709 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:12.776750 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:12.779591 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:12.780381 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:12.780389 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:12.780391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:12.780539 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:12.780866 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:12.780870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:12.780872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:12.781257 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:12.781512 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:12.781783 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 20:29:12.782022 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":257,"threads_started":1}
W20260929 20:29:12.783681 584 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47514
W20260929 20:29:12.784862 29650 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 461: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:29:12.785020 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 461
I20260929 20:29:12.801610 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 595
Sep 29 20:29:12 dist-test-slave-p29l krb5kdc[32689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713752, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.244.131@KRBTEST.COM
I20260929 20:29:12.818284 737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:12.802608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39488 (local address 127.28.244.131:35797)
0929 20:29:12.803282 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:12.803289 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:12.803371 (+ 82us) server_negotiation.cc:407] Connection header received
0929 20:29:12.803457 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:12.803464 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:12.803701 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:12.803889 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:12.805229 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:12.806284 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:12.807540 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:12.808185 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:12.808288 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:12.812654 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:12.812668 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:12.812685 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:12.812728 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:12.815544 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:12.816392 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:12.816400 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:12.816403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:12.816547 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:12.816890 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:12.816893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:12.816895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:12.817258 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:12.817495 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:12.817761 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 20:29:12.817974 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":328,"threads_started":1}
W20260929 20:29:12.819733 718 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39488
W20260929 20:29:12.821028 29650 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 595: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:29:12.821209 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 595
I20260929 20:29:12.837317 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 32702
I20260929 20:29:12.838657 32764 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:29:12.943638 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 32702
2026-09-29T20:29:12Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3744 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:29:13 dist-test-slave-p29l krb5kdc[741](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:29:13 dist-test-slave-p29l krb5kdc[741](info): set up 2 sockets
Sep 29 20:29:13 dist-test-slave-p29l krb5kdc[741](info): commencing operation
krb5kdc: starting...
W20260929 20:29:14.076704 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:29:14 dist-test-slave-p29l krb5kdc[741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:29:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:29:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.244.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
I20260929 20:29:14.305963 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:43249
--webserver_interface=127.28.244.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:35323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:43249
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:29:14.671517 754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:14.671947 754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:14.672070 754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:14.679693 754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:29:14.679896 754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:14.679966 754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:14.680028 754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:29:14.680073 754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:29:14.690574 754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:35323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:43249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:43249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:14.692059 754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:14.693506 754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:14.706792 759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:14.706848 760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:14.706770 762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:14.708087 754 server_base.cc:1034] running on GCE node
I20260929 20:29:14.709000 754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:14.710610 754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:14.711910 754 hybrid_clock.cc:648] HybridClock initialized: now 1790713754711831 us; error 61 us; skew 500 ppm
I20260929 20:29:14.712337 754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:14 dist-test-slave-p29l krb5kdc[741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713754, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:14.722246 754 init.cc:377] Logged in from keytab as kudu/127.28.244.190@KRBTEST.COM (short username kudu)
I20260929 20:29:14.724231 754 webserver.cc:460] Webserver started at http://127.28.244.190:41971/ using document root <none> and password file <none>
I20260929 20:29:14.725088 754 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:14.725235 754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:14.725605 754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:14.729701 754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "57e6bd8b72944f2b8102b7d3173de059"
format_stamp: "Formatted at 2026-09-29 20:29:14 on dist-test-slave-p29l"
server_key: "1d494e1d034f7e594323429676da30c3"
server_key_iv: "bd82c69708f07a06fe9a758b9ff05dfb"
server_key_version: "encryptionkey@0"
I20260929 20:29:14.731096 754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "57e6bd8b72944f2b8102b7d3173de059"
format_stamp: "Formatted at 2026-09-29 20:29:14 on dist-test-slave-p29l"
server_key: "1d494e1d034f7e594323429676da30c3"
server_key_iv: "bd82c69708f07a06fe9a758b9ff05dfb"
server_key_version: "encryptionkey@0"
I20260929 20:29:14.740056 754 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20260929 20:29:14.744318 769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:14.746006 754 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.001s
I20260929 20:29:14.746253 754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "57e6bd8b72944f2b8102b7d3173de059"
format_stamp: "Formatted at 2026-09-29 20:29:14 on dist-test-slave-p29l"
server_key: "1d494e1d034f7e594323429676da30c3"
server_key_iv: "bd82c69708f07a06fe9a758b9ff05dfb"
server_key_version: "encryptionkey@0"
I20260929 20:29:14.746510 754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:14.763702 754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:14.768800 754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:14.769127 754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:14.789562 754 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:43249
I20260929 20:29:14.789595 820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:43249 every 8 connection(s)
I20260929 20:29:14.791265 754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:29:14.795801 821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:14.799409 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 754
I20260929 20:29:14.799687 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:29:14.800450 29650 external_mini_cluster.cc:1428] Setting key 3763643729655473690968bc5cf01ae9
I20260929 20:29:14.810721 821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57e6bd8b72944f2b8102b7d3173de059: Bootstrap starting.
I20260929 20:29:14.815368 821 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57e6bd8b72944f2b8102b7d3173de059: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:14.817063 821 log.cc:824] T 00000000000000000000000000000000 P 57e6bd8b72944f2b8102b7d3173de059: Log is configured to *not* fsync() on all Append() calls
Sep 29 20:29:14 dist-test-slave-p29l krb5kdc[741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:29:14.820982 821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57e6bd8b72944f2b8102b7d3173de059: No bootstrap required, opened a new log
I20260929 20:29:14.828253 824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:14.802921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45970 (local address 127.28.244.190:43249)
0929 20:29:14.803911 (+ 990us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:14.803924 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:14.803971 (+ 47us) server_negotiation.cc:407] Connection header received
0929 20:29:14.805761 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:14.805863 (+ 102us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:14.806546 (+ 683us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:14.806976 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:14.809079 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:14.810570 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:14.812506 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:14.813445 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:14.813618 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:14.819408 (+ 5790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:14.819420 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:14.819440 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:14.819486 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:14.823188 (+ 3702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:14.824164 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:14.824176 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:14.824184 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:14.824400 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:14.824844 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:14.824849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:14.824851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:14.825571 (+ 720us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:14.825912 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:14.826482 (+ 570us) server_negotiation.cc:299] Negotiation successful
0929 20:29:14.826982 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":458,"threads_started":1}
I20260929 20:29:14.831334 821 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57e6bd8b72944f2b8102b7d3173de059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e6bd8b72944f2b8102b7d3173de059" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 43249 } }
I20260929 20:29:14.832029 821 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57e6bd8b72944f2b8102b7d3173de059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:14.832181 821 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57e6bd8b72944f2b8102b7d3173de059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57e6bd8b72944f2b8102b7d3173de059, State: Initialized, Role: FOLLOWER
I20260929 20:29:14.832891 821 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57e6bd8b72944f2b8102b7d3173de059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e6bd8b72944f2b8102b7d3173de059" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 43249 } }
I20260929 20:29:14.833177 821 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57e6bd8b72944f2b8102b7d3173de059 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:29:14.833354 821 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57e6bd8b72944f2b8102b7d3173de059 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:29:14.833567 821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57e6bd8b72944f2b8102b7d3173de059 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:14.836582 821 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57e6bd8b72944f2b8102b7d3173de059 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e6bd8b72944f2b8102b7d3173de059" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 43249 } }
I20260929 20:29:14.837329 821 leader_election.cc:302] T 00000000000000000000000000000000 P 57e6bd8b72944f2b8102b7d3173de059 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57e6bd8b72944f2b8102b7d3173de059; no voters:
I20260929 20:29:14.838054 821 leader_election.cc:288] T 00000000000000000000000000000000 P 57e6bd8b72944f2b8102b7d3173de059 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:29:14.838246 826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57e6bd8b72944f2b8102b7d3173de059 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:29:14.839197 826 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57e6bd8b72944f2b8102b7d3173de059 [term 1 LEADER]: Becoming Leader. State: Replica: 57e6bd8b72944f2b8102b7d3173de059, State: Running, Role: LEADER
I20260929 20:29:14.839895 826 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57e6bd8b72944f2b8102b7d3173de059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e6bd8b72944f2b8102b7d3173de059" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 43249 } }
I20260929 20:29:14.840807 821 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57e6bd8b72944f2b8102b7d3173de059 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:29:14.842876 827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57e6bd8b72944f2b8102b7d3173de059 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57e6bd8b72944f2b8102b7d3173de059" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e6bd8b72944f2b8102b7d3173de059" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 43249 } } }
I20260929 20:29:14.843604 827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57e6bd8b72944f2b8102b7d3173de059 [sys.catalog]: This master's current role is: LEADER
I20260929 20:29:14.843793 828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57e6bd8b72944f2b8102b7d3173de059 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57e6bd8b72944f2b8102b7d3173de059. Latest consensus state: current_term: 1 leader_uuid: "57e6bd8b72944f2b8102b7d3173de059" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e6bd8b72944f2b8102b7d3173de059" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 43249 } } }
I20260929 20:29:14.844786 828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57e6bd8b72944f2b8102b7d3173de059 [sys.catalog]: This master's current role is: LEADER
I20260929 20:29:14.849511 832 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:29:14.855404 832 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:29:14.866324 832 catalog_manager.cc:1347] Generated new cluster ID: 072b58c8623945a4957ae4d82e0d2ded
I20260929 20:29:14.866469 832 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:29:14.880319 832 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:29:14.882489 832 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:29:14.900388 832 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57e6bd8b72944f2b8102b7d3173de059: Generated new TSK 0
I20260929 20:29:14.901445 832 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.244.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab.
Entry for principal kudu/127.28.244.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab.
I20260929 20:29:14.965549 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.129:0
--local_ip_for_outbound_sockets=127.28.244.129
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:43249
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:35323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:29:15.341246 847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:15.341612 847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:15.341795 847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:15.349798 847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:15.349972 847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:15.350180 847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.129
I20260929 20:29:15.360798 847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:35323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:43249
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.129
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:15.362155 847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:15.363608 847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:15.379086 853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:15.379166 855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:15.379180 852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:15.380082 847 server_base.cc:1034] running on GCE node
I20260929 20:29:15.380782 847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:15.381955 847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:15.383203 847 hybrid_clock.cc:648] HybridClock initialized: now 1790713755383123 us; error 75 us; skew 500 ppm
I20260929 20:29:15.383538 847 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:15 dist-test-slave-p29l krb5kdc[741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713755, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:15.388060 847 init.cc:377] Logged in from keytab as kudu/127.28.244.129@KRBTEST.COM (short username kudu)
I20260929 20:29:15.393987 847 webserver.cc:460] Webserver started at http://127.28.244.129:42535/ using document root <none> and password file <none>
I20260929 20:29:15.394941 847 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:15.395094 847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:15.395442 847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:15.398779 847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-0/data/instance:
uuid: "338201b2fa3e4845bc96f84ffd308423"
format_stamp: "Formatted at 2026-09-29 20:29:15 on dist-test-slave-p29l"
server_key: "a19f4a2909c49e7eac17b4ec8b8257ba"
server_key_iv: "a0725c8777b3adb71a66cd961047b1a9"
server_key_version: "encryptionkey@0"
I20260929 20:29:15.399696 847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance:
uuid: "338201b2fa3e4845bc96f84ffd308423"
format_stamp: "Formatted at 2026-09-29 20:29:15 on dist-test-slave-p29l"
server_key: "a19f4a2909c49e7eac17b4ec8b8257ba"
server_key_iv: "a0725c8777b3adb71a66cd961047b1a9"
server_key_version: "encryptionkey@0"
I20260929 20:29:15.405781 847 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 20:29:15.410167 862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:15.411957 847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:29:15.412174 847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-0/wal
uuid: "338201b2fa3e4845bc96f84ffd308423"
format_stamp: "Formatted at 2026-09-29 20:29:15 on dist-test-slave-p29l"
server_key: "a19f4a2909c49e7eac17b4ec8b8257ba"
server_key_iv: "a0725c8777b3adb71a66cd961047b1a9"
server_key_version: "encryptionkey@0"
I20260929 20:29:15.412436 847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:15.432338 847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:15.437417 847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:15.437755 847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:15.438881 847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:29:15.440572 847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:29:15.440678 847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:15.440955 847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:29:15.441052 847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:15.480346 847 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.129:42245
I20260929 20:29:15.480381 974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.129:42245 every 8 connection(s)
I20260929 20:29:15.481904 847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
I20260929 20:29:15.485080 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 847
I20260929 20:29:15.485334 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance
I20260929 20:29:15.486092 29650 external_mini_cluster.cc:1428] Setting key 8bb5600323eeb454863d9ec6a1a87d90
Sep 29 20:29:15 dist-test-slave-p29l krb5kdc[741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713755, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.244.130@KRBTEST.COM; defaulting to no policy
I20260929 20:29:15.506227 977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:15.485102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:44451 (local address 127.28.244.190:43249)
0929 20:29:15.485717 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:15.485724 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:15.487002 (+ 1278us) server_negotiation.cc:407] Connection header received
0929 20:29:15.489563 (+ 2561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:15.489569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:15.489670 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:15.489792 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:15.491652 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:15.492482 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:15.494542 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:15.494993 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:15.495144 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:15.499439 (+ 4295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:15.499445 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:15.499448 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:15.499496 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:15.502560 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:15.503480 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:15.503488 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:15.503492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:15.503633 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:15.504193 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:15.504199 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:15.504207 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:15.504505 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:15.504668 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:15.505577 (+ 909us) server_negotiation.cc:299] Negotiation successful
0929 20:29:15.505845 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":308,"threads_started":1}
I20260929 20:29:15.507408 978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:15.485834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:43249 (local address 127.28.244.129:44451)
0929 20:29:15.486787 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:15.486859 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:15.489210 (+ 2351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:15.490096 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:15.490106 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:15.490715 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:15.491434 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:15.491455 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:15.492697 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:15.492701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:15.494282 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:15.494349 (+ 67us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:15.495273 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:15.495281 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:15.495546 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:15.496424 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:15.496455 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:15.499133 (+ 2678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:15.502784 (+ 3651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:15.502801 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:15.502819 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:15.503337 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:15.503757 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:15.503763 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:15.503767 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:15.504069 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:15.504655 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:15.504664 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:15.505123 (+ 459us) client_negotiation.cc:769] Sending connection context
0929 20:29:15.505453 (+ 330us) client_negotiation.cc:240] Negotiation successful
0929 20:29:15.506080 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":357,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.130@KRBTEST.COM" created.
I20260929 20:29:15.509240 975 heartbeater.cc:347] Connected to a master server at 127.28.244.190:43249
I20260929 20:29:15.509596 975 heartbeater.cc:464] Registering TS with master...
I20260929 20:29:15.510646 975 heartbeater.cc:511] Master 127.28.244.190:43249 requested a full tablet report, sending...
I20260929 20:29:15.513360 786 ts_manager.cc:194] Registered new tserver with Master: 338201b2fa3e4845bc96f84ffd308423 (127.28.244.129:42245)
I20260929 20:29:15.515885 786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.129@KRBTEST.COM'} at 127.28.244.129:44451
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab.
Entry for principal kudu/127.28.244.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab.
I20260929 20:29:15.535568 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.130:0
--local_ip_for_outbound_sockets=127.28.244.130
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:43249
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:35323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:29:15.902132 981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:15.902460 981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:15.902621 981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:15.910414 981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:15.910547 981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:15.910734 981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.130
I20260929 20:29:15.920945 981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:35323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:43249
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.130
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:15.922377 981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:15.923836 981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:15.938534 986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:15.938481 989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:15.939033 987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:15.939316 981 server_base.cc:1034] running on GCE node
I20260929 20:29:15.940075 981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:15.941329 981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:15.942581 981 hybrid_clock.cc:648] HybridClock initialized: now 1790713755942517 us; error 61 us; skew 500 ppm
I20260929 20:29:15.942965 981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:15 dist-test-slave-p29l krb5kdc[741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713755, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:15.946393 981 init.cc:377] Logged in from keytab as kudu/127.28.244.130@KRBTEST.COM (short username kudu)
I20260929 20:29:15.948122 981 webserver.cc:460] Webserver started at http://127.28.244.130:38983/ using document root <none> and password file <none>
I20260929 20:29:15.948853 981 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:15.948954 981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:15.949221 981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:15.952565 981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-1/data/instance:
uuid: "943d8cd8a13541189fffbcdd89cf9b87"
format_stamp: "Formatted at 2026-09-29 20:29:15 on dist-test-slave-p29l"
server_key: "1208c09e70782efb197da5f708c7d1f5"
server_key_iv: "a6bb8014b25727aa8a6833451b9717e2"
server_key_version: "encryptionkey@0"
I20260929 20:29:15.953380 981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance:
uuid: "943d8cd8a13541189fffbcdd89cf9b87"
format_stamp: "Formatted at 2026-09-29 20:29:15 on dist-test-slave-p29l"
server_key: "1208c09e70782efb197da5f708c7d1f5"
server_key_iv: "a6bb8014b25727aa8a6833451b9717e2"
server_key_version: "encryptionkey@0"
I20260929 20:29:15.959182 981 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:29:15.963373 996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:15.965133 981 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 20:29:15.965332 981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-1/wal
uuid: "943d8cd8a13541189fffbcdd89cf9b87"
format_stamp: "Formatted at 2026-09-29 20:29:15 on dist-test-slave-p29l"
server_key: "1208c09e70782efb197da5f708c7d1f5"
server_key_iv: "a6bb8014b25727aa8a6833451b9717e2"
server_key_version: "encryptionkey@0"
I20260929 20:29:15.965569 981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:15.978554 981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:15.983042 981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:15.983356 981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:15.984547 981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:29:15.985980 981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:29:15.986055 981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:15.986197 981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:29:15.986248 981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:16.021768 981 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.130:40805
I20260929 20:29:16.021826 1108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.130:40805 every 8 connection(s)
I20260929 20:29:16.023172 981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
I20260929 20:29:16.027050 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 981
I20260929 20:29:16.027262 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance
I20260929 20:29:16.027758 29650 external_mini_cluster.cc:1428] Setting key 3822eab45a5204d133578fdd22edfbdf
Sep 29 20:29:16 dist-test-slave-p29l krb5kdc[741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713755, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:29:16.048345 1111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:16.026068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:39467 (local address 127.28.244.190:43249)
0929 20:29:16.026523 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:16.026528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:16.027732 (+ 1204us) server_negotiation.cc:407] Connection header received
0929 20:29:16.030022 (+ 2290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:16.030027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:16.030111 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:16.030227 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:16.031880 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:16.033039 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:16.034691 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:16.035414 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:16.035518 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:16.040294 (+ 4776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:16.040304 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:16.040312 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:16.040364 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:16.044400 (+ 4036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:16.045379 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:16.045388 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:16.045395 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:16.045596 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:16.046095 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:16.046102 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:16.046107 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:16.046545 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:16.046717 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:16.047583 (+ 866us) server_negotiation.cc:299] Negotiation successful
0929 20:29:16.047886 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":217,"threads_started":1}
WARNING: no policy specified for kudu/127.28.244.131@KRBTEST.COM; defaulting to no policy
I20260929 20:29:16.049256 1112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:16.026688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:43249 (local address 127.28.244.130:39467)
0929 20:29:16.027542 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:16.027595 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:16.029668 (+ 2073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:16.030444 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:16.030452 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:16.031027 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:16.031653 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:16.031671 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:16.033225 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:16.033229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:16.034531 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:16.034548 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:16.035693 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:16.035703 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:16.036076 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:16.036908 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:16.036943 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:16.040007 (+ 3064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:16.044620 (+ 4613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:16.044633 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:16.044657 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:16.045225 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:16.045734 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:16.045737 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:16.045740 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:16.045947 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:16.046713 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:16.046721 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:16.047123 (+ 402us) client_negotiation.cc:769] Sending connection context
0929 20:29:16.047438 (+ 315us) client_negotiation.cc:240] Negotiation successful
0929 20:29:16.047971 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":375,"threads_started":1}
I20260929 20:29:16.051016 1109 heartbeater.cc:347] Connected to a master server at 127.28.244.190:43249
I20260929 20:29:16.051403 1109 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.131@KRBTEST.COM" created.
I20260929 20:29:16.052536 1109 heartbeater.cc:511] Master 127.28.244.190:43249 requested a full tablet report, sending...
I20260929 20:29:16.054486 786 ts_manager.cc:194] Registered new tserver with Master: 943d8cd8a13541189fffbcdd89cf9b87 (127.28.244.130:40805)
I20260929 20:29:16.056012 786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.130@KRBTEST.COM'} at 127.28.244.130:39467
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab.
Entry for principal kudu/127.28.244.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab.
I20260929 20:29:16.080185 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.131:0
--local_ip_for_outbound_sockets=127.28.244.131
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:43249
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:35323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:29:16.429430 1115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:16.429759 1115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:16.429932 1115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:16.437395 1115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:16.437582 1115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:16.437755 1115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.131
I20260929 20:29:16.448065 1115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:35323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:43249
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.131
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:16.449592 1115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:16.451234 1115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:16.466820 1121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:16.466885 1123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:16.466854 1120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:16.467664 1115 server_base.cc:1034] running on GCE node
I20260929 20:29:16.468431 1115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:16.469642 1115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:16.470903 1115 hybrid_clock.cc:648] HybridClock initialized: now 1790713756470829 us; error 71 us; skew 500 ppm
I20260929 20:29:16.471374 1115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:16 dist-test-slave-p29l krb5kdc[741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713756, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:16.474943 1115 init.cc:377] Logged in from keytab as kudu/127.28.244.131@KRBTEST.COM (short username kudu)
I20260929 20:29:16.476662 1115 webserver.cc:460] Webserver started at http://127.28.244.131:36265/ using document root <none> and password file <none>
I20260929 20:29:16.477428 1115 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:16.477568 1115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:16.477887 1115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:16.481361 1115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-2/data/instance:
uuid: "6d49e34bdf854e29af94219d6dbc2eb4"
format_stamp: "Formatted at 2026-09-29 20:29:16 on dist-test-slave-p29l"
server_key: "1ecbffdf4ba7abd344efccf93ef8330a"
server_key_iv: "61dd7f3afc438cbea2f331ca63cb7d5c"
server_key_version: "encryptionkey@0"
I20260929 20:29:16.482249 1115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance:
uuid: "6d49e34bdf854e29af94219d6dbc2eb4"
format_stamp: "Formatted at 2026-09-29 20:29:16 on dist-test-slave-p29l"
server_key: "1ecbffdf4ba7abd344efccf93ef8330a"
server_key_iv: "61dd7f3afc438cbea2f331ca63cb7d5c"
server_key_version: "encryptionkey@0"
I20260929 20:29:16.488217 1115 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260929 20:29:16.492233 1130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:16.493968 1115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:29:16.494195 1115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-2/wal
uuid: "6d49e34bdf854e29af94219d6dbc2eb4"
format_stamp: "Formatted at 2026-09-29 20:29:16 on dist-test-slave-p29l"
server_key: "1ecbffdf4ba7abd344efccf93ef8330a"
server_key_iv: "61dd7f3afc438cbea2f331ca63cb7d5c"
server_key_version: "encryptionkey@0"
I20260929 20:29:16.494454 1115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:16.519912 975 heartbeater.cc:503] Master 127.28.244.190:43249 was elected leader, sending a full tablet report...
I20260929 20:29:16.525192 1115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:16.530072 1115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:16.530440 1115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:16.531549 1115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:29:16.533107 1115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:29:16.533217 1115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:16.533368 1115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:29:16.533437 1115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:16.568225 1115 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.131:42041
I20260929 20:29:16.568288 1242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.131:42041 every 8 connection(s)
I20260929 20:29:16.569667 1115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
I20260929 20:29:16.571455 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 1115
I20260929 20:29:16.571664 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance
I20260929 20:29:16.572240 29650 external_mini_cluster.cc:1428] Setting key 34e1d5f5618d81f96ec5e6d314d21920
Sep 29 20:29:16 dist-test-slave-p29l krb5kdc[741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713756, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:29:16.593411 1245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:16.573104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:33357 (local address 127.28.244.190:43249)
0929 20:29:16.573850 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:16.573856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:16.575067 (+ 1211us) server_negotiation.cc:407] Connection header received
0929 20:29:16.577308 (+ 2241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:16.577313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:16.577389 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:16.577498 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:16.579259 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:16.580244 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:16.582012 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:16.582489 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:16.582606 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:16.586593 (+ 3987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:16.586601 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:16.586612 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:16.586659 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:16.590017 (+ 3358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:16.590858 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:16.590862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:16.590864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:16.590993 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:16.591480 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:16.591483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:16.591485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:16.591749 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:16.591941 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:16.592713 (+ 772us) server_negotiation.cc:299] Negotiation successful
0929 20:29:16.592949 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":441,"threads_started":1}
I20260929 20:29:16.594600 1246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:16.573907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:43249 (local address 127.28.244.131:33357)
0929 20:29:16.574860 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:16.574940 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:16.576982 (+ 2042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:16.577775 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:16.577785 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:16.578403 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:16.579071 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:16.579091 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:16.580419 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:16.580424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:16.581804 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:16.581825 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:16.582735 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:16.582741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:16.582991 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:16.583737 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:16.583762 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:16.586310 (+ 2548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:16.590181 (+ 3871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:16.590189 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:16.590206 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:16.590737 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:16.591111 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:16.591114 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:16.591117 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:16.591382 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:16.591938 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:16.591945 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:16.592351 (+ 406us) client_negotiation.cc:769] Sending connection context
0929 20:29:16.592624 (+ 273us) client_negotiation.cc:240] Negotiation successful
0929 20:29:16.593258 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":369,"threads_started":1}
I20260929 20:29:16.596122 1243 heartbeater.cc:347] Connected to a master server at 127.28.244.190:43249
I20260929 20:29:16.596474 1243 heartbeater.cc:464] Registering TS with master...
I20260929 20:29:16.597254 1243 heartbeater.cc:511] Master 127.28.244.190:43249 requested a full tablet report, sending...
I20260929 20:29:16.598855 786 ts_manager.cc:194] Registered new tserver with Master: 6d49e34bdf854e29af94219d6dbc2eb4 (127.28.244.131:42041)
I20260929 20:29:16.599938 786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.131@KRBTEST.COM'} at 127.28.244.131:33357
I20260929 20:29:16.601451 29650 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 20:29:16 dist-test-slave-p29l krb5kdc[741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 20:29:16 dist-test-slave-p29l krb5kdc[741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.129@KRBTEST.COM
I20260929 20:29:16.657473 1251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:16.639983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49096 (local address 127.28.244.129:42245)
0929 20:29:16.640671 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:16.640678 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:16.640707 (+ 29us) server_negotiation.cc:407] Connection header received
0929 20:29:16.640770 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:16.640777 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:16.641046 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:16.641188 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:16.642922 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:16.644071 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:16.645494 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:16.646123 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:16.646246 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:16.650822 (+ 4576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:16.650839 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:16.650857 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:16.650929 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:16.654098 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:16.654979 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:16.654987 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:16.654990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:16.655145 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:16.655610 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:16.655614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:16.655616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:16.656095 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:16.656343 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:16.656768 (+ 425us) server_negotiation.cc:299] Negotiation successful
0929 20:29:16.657009 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":292,"threads_started":1}
Sep 29 20:29:16 dist-test-slave-p29l krb5kdc[741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:29:16.679060 1245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:16.662773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45972 (local address 127.28.244.190:43249)
0929 20:29:16.663039 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:16.663045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:16.663252 (+ 207us) server_negotiation.cc:407] Connection header received
0929 20:29:16.663405 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:16.663409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:16.663523 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:16.663609 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:16.665130 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:16.665930 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:16.667369 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:16.667783 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:16.667957 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:16.672770 (+ 4813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:16.672776 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:16.672781 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:16.672828 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:16.675698 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:16.676576 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:16.676580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:16.676583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:16.676727 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:16.677168 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:16.677172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:16.677174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:16.677779 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:16.678009 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:16.678422 (+ 413us) server_negotiation.cc:299] Negotiation successful
0929 20:29:16.678652 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
W20260929 20:29:16.679720 786 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45972
I20260929 20:29:16.680963 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 847
I20260929 20:29:16.694808 1251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:16.682108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49102 (local address 127.28.244.129:42245)
0929 20:29:16.682325 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:16.682332 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:16.682471 (+ 139us) server_negotiation.cc:407] Connection header received
0929 20:29:16.682620 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:16.682624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:16.682719 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:16.682803 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:16.684105 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:16.684809 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:16.686035 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:16.686518 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:16.686621 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:16.689312 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:16.689317 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:16.689321 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:16.689358 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:16.692156 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:16.692954 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:16.692959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:16.692961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:16.693100 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:16.693512 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:16.693515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:16.693517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:16.693785 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:16.693971 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:16.694295 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 20:29:16.694484 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 20:29:16.695577 970 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:29:16.781010 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 847
I20260929 20:29:16.798354 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 981
Sep 29 20:29:16 dist-test-slave-p29l krb5kdc[741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.130@KRBTEST.COM
I20260929 20:29:16.817381 1256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:16.799536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41740 (local address 127.28.244.130:40805)
0929 20:29:16.800355 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:16.800363 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:16.800389 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:29:16.800453 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:16.800459 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:16.800689 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:16.800821 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:16.802051 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:16.803089 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:16.804583 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:16.805243 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:16.805372 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:16.810373 (+ 5001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:16.810392 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:16.810410 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:16.810459 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:16.813895 (+ 3436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:16.814875 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:16.814885 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:16.814888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:16.815068 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:16.815610 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:16.815615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:16.815618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:16.816135 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:16.816405 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:16.816832 (+ 427us) server_negotiation.cc:299] Negotiation successful
0929 20:29:16.817075 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":500,"threads_started":1}
I20260929 20:29:16.818948 1104 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:29:16.902509 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 981
I20260929 20:29:16.920969 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 1115
Sep 29 20:29:16 dist-test-slave-p29l krb5kdc[741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.131@KRBTEST.COM
I20260929 20:29:16.938576 1258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:16.922015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44212 (local address 127.28.244.131:42041)
0929 20:29:16.922620 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:16.922627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:16.922658 (+ 31us) server_negotiation.cc:407] Connection header received
0929 20:29:16.922721 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:16.922728 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:16.922956 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:16.923114 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:16.924513 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:16.925597 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:16.926904 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:16.927567 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:16.927690 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:16.932354 (+ 4664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:16.932372 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:16.932390 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:16.932441 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:16.935449 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:16.936379 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:16.936389 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:16.936391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:16.936538 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:16.936933 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:16.936936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:16.936938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:16.937327 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:16.937570 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:16.937920 (+ 350us) server_negotiation.cc:299] Negotiation successful
0929 20:29:16.938221 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":282,"threads_started":1}
I20260929 20:29:16.940119 1238 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:29:17.027367 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 1115
I20260929 20:29:17.044674 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 754
I20260929 20:29:17.046131 816 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:29:17.155967 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 754
2026-09-29T20:29:17Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4211 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:29:17 dist-test-slave-p29l krb5kdc[1263](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:29:17 dist-test-slave-p29l krb5kdc[1263](info): set up 2 sockets
Sep 29 20:29:17 dist-test-slave-p29l krb5kdc[1263](info): commencing operation
krb5kdc: starting...
W20260929 20:29:18.328187 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:29:18 dist-test-slave-p29l krb5kdc[1263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:29:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:29:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.244.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
I20260929 20:29:18.564668 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:42573
--webserver_interface=127.28.244.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:44529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:42573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:29:18.936165 1276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:18.936534 1276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:18.936720 1276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:18.944772 1276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:29:18.944921 1276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:18.945029 1276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:18.945116 1276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:29:18.945185 1276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:29:18.955830 1276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:44529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:42573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:42573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:18.957274 1276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:18.958735 1276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:18.971660 1281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:18.971591 1282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:18.971591 1284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:18.972918 1276 server_base.cc:1034] running on GCE node
I20260929 20:29:18.973879 1276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:18.975580 1276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:18.976856 1276 hybrid_clock.cc:648] HybridClock initialized: now 1790713758976798 us; error 61 us; skew 500 ppm
I20260929 20:29:18.977280 1276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:18 dist-test-slave-p29l krb5kdc[1263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713758, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:18.980988 1276 init.cc:377] Logged in from keytab as kudu/127.28.244.190@KRBTEST.COM (short username kudu)
I20260929 20:29:18.982402 1276 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:18.982549 1276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:18.982906 1276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:18.986235 1276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "95843243b469455c9d092e0511efab91"
format_stamp: "Formatted at 2026-09-29 20:29:18 on dist-test-slave-p29l"
server_key: "7646e993ec0ffe373e70ac93ee323b52"
server_key_iv: "fe3c4766d36ea2c7e5b524d0bf9131e6"
server_key_version: "encryptionkey@0"
I20260929 20:29:18.987159 1276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "95843243b469455c9d092e0511efab91"
format_stamp: "Formatted at 2026-09-29 20:29:18 on dist-test-slave-p29l"
server_key: "7646e993ec0ffe373e70ac93ee323b52"
server_key_iv: "fe3c4766d36ea2c7e5b524d0bf9131e6"
server_key_version: "encryptionkey@0"
I20260929 20:29:18.993643 1276 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.002s
I20260929 20:29:18.998026 1290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:18.999758 1276 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260929 20:29:19.000074 1276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "95843243b469455c9d092e0511efab91"
format_stamp: "Formatted at 2026-09-29 20:29:18 on dist-test-slave-p29l"
server_key: "7646e993ec0ffe373e70ac93ee323b52"
server_key_iv: "fe3c4766d36ea2c7e5b524d0bf9131e6"
server_key_version: "encryptionkey@0"
I20260929 20:29:19.000341 1276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:19.023260 1276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:19.028412 1276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:19.028717 1276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:19.049374 1341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:42573 every 8 connection(s)
I20260929 20:29:19.049487 1276 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:42573
I20260929 20:29:19.050506 1276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:29:19.054827 1342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:19.059226 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 1276
I20260929 20:29:19.059607 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:29:19.060563 29650 external_mini_cluster.cc:1428] Setting key 5c6cc3b9c625d41d145a86b9c4181178
I20260929 20:29:19.069092 1342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95843243b469455c9d092e0511efab91: Bootstrap starting.
I20260929 20:29:19.072695 1342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95843243b469455c9d092e0511efab91: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:19.073872 1342 log.cc:824] T 00000000000000000000000000000000 P 95843243b469455c9d092e0511efab91: Log is configured to *not* fsync() on all Append() calls
I20260929 20:29:19.077358 1342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95843243b469455c9d092e0511efab91: No bootstrap required, opened a new log
Sep 29 20:29:19 dist-test-slave-p29l krb5kdc[1263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:29:19.087522 1345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:19.063470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57142 (local address 127.28.244.190:42573)
0929 20:29:19.064447 (+ 977us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:19.064461 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:19.064509 (+ 48us) server_negotiation.cc:407] Connection header received
0929 20:29:19.066895 (+ 2386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:19.066931 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:19.067462 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:19.068181 (+ 719us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:19.069972 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:19.071445 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:19.072980 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:19.073756 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:19.073992 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:19.078820 (+ 4828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:19.078828 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:19.078846 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:19.078894 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:19.082543 (+ 3649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:19.083582 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:19.083591 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:19.083602 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:19.083879 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:19.084393 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:19.084398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:19.084400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:19.085083 (+ 683us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:19.085364 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:19.085881 (+ 517us) server_negotiation.cc:299] Negotiation successful
0929 20:29:19.086343 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":433,"threads_started":1}
I20260929 20:29:19.087544 1342 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95843243b469455c9d092e0511efab91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95843243b469455c9d092e0511efab91" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 42573 } }
I20260929 20:29:19.088128 1342 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95843243b469455c9d092e0511efab91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:19.088246 1342 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95843243b469455c9d092e0511efab91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95843243b469455c9d092e0511efab91, State: Initialized, Role: FOLLOWER
I20260929 20:29:19.088840 1342 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95843243b469455c9d092e0511efab91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95843243b469455c9d092e0511efab91" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 42573 } }
I20260929 20:29:19.089097 1342 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95843243b469455c9d092e0511efab91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:29:19.089249 1342 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95843243b469455c9d092e0511efab91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:29:19.089457 1342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95843243b469455c9d092e0511efab91 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:19.092000 1342 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95843243b469455c9d092e0511efab91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95843243b469455c9d092e0511efab91" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 42573 } }
I20260929 20:29:19.092592 1342 leader_election.cc:302] T 00000000000000000000000000000000 P 95843243b469455c9d092e0511efab91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95843243b469455c9d092e0511efab91; no voters:
I20260929 20:29:19.093209 1342 leader_election.cc:288] T 00000000000000000000000000000000 P 95843243b469455c9d092e0511efab91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:29:19.093385 1347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95843243b469455c9d092e0511efab91 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:29:19.094142 1347 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95843243b469455c9d092e0511efab91 [term 1 LEADER]: Becoming Leader. State: Replica: 95843243b469455c9d092e0511efab91, State: Running, Role: LEADER
I20260929 20:29:19.094726 1347 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95843243b469455c9d092e0511efab91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95843243b469455c9d092e0511efab91" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 42573 } }
I20260929 20:29:19.095479 1342 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95843243b469455c9d092e0511efab91 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:29:19.097889 1348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95843243b469455c9d092e0511efab91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95843243b469455c9d092e0511efab91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95843243b469455c9d092e0511efab91" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 42573 } } }
I20260929 20:29:19.098282 1348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95843243b469455c9d092e0511efab91 [sys.catalog]: This master's current role is: LEADER
I20260929 20:29:19.098276 1349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95843243b469455c9d092e0511efab91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95843243b469455c9d092e0511efab91. Latest consensus state: current_term: 1 leader_uuid: "95843243b469455c9d092e0511efab91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95843243b469455c9d092e0511efab91" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 42573 } } }
I20260929 20:29:19.098796 1349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95843243b469455c9d092e0511efab91 [sys.catalog]: This master's current role is: LEADER
I20260929 20:29:19.101900 1353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:29:19.108763 1353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:29:19.121820 1353 catalog_manager.cc:1347] Generated new cluster ID: 9440213a3d40422ca40d4dfa80b54b92
I20260929 20:29:19.121977 1353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:29:19.132817 1353 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:29:19.135066 1353 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:29:19.143177 1353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95843243b469455c9d092e0511efab91: Generated new TSK 0
I20260929 20:29:19.144294 1353 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.244.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab.
Entry for principal kudu/127.28.244.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab.
I20260929 20:29:19.205463 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.129:0
--local_ip_for_outbound_sockets=127.28.244.129
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:42573
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:44529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:29:19.589965 1368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:19.590273 1368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:19.590384 1368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:19.598166 1368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:19.598289 1368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:19.598417 1368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.129
I20260929 20:29:19.609543 1368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:44529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:42573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.129
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:19.610927 1368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:19.612433 1368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:19.627573 1373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:19.627444 1374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:19.627736 1376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:19.629442 1368 server_base.cc:1034] running on GCE node
I20260929 20:29:19.630252 1368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:19.631449 1368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:19.632753 1368 hybrid_clock.cc:648] HybridClock initialized: now 1790713759632695 us; error 83 us; skew 500 ppm
I20260929 20:29:19.633170 1368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:19 dist-test-slave-p29l krb5kdc[1263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713759, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:19.636829 1368 init.cc:377] Logged in from keytab as kudu/127.28.244.129@KRBTEST.COM (short username kudu)
I20260929 20:29:19.638322 1368 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:19.638480 1368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:19.638856 1368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:19.642351 1368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-0/data/instance:
uuid: "ec38e974d7114bb7aa05c1bd69a221a2"
format_stamp: "Formatted at 2026-09-29 20:29:19 on dist-test-slave-p29l"
server_key: "798aacd9c61eab0bb5122c922dbde5f7"
server_key_iv: "0bcc6f4425125350274999e7153e20fa"
server_key_version: "encryptionkey@0"
I20260929 20:29:19.643530 1368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance:
uuid: "ec38e974d7114bb7aa05c1bd69a221a2"
format_stamp: "Formatted at 2026-09-29 20:29:19 on dist-test-slave-p29l"
server_key: "798aacd9c61eab0bb5122c922dbde5f7"
server_key_iv: "0bcc6f4425125350274999e7153e20fa"
server_key_version: "encryptionkey@0"
I20260929 20:29:19.650947 1368 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.005s
I20260929 20:29:19.656854 1382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:19.660018 1368 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.001s
I20260929 20:29:19.660477 1368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-0/wal
uuid: "ec38e974d7114bb7aa05c1bd69a221a2"
format_stamp: "Formatted at 2026-09-29 20:29:19 on dist-test-slave-p29l"
server_key: "798aacd9c61eab0bb5122c922dbde5f7"
server_key_iv: "0bcc6f4425125350274999e7153e20fa"
server_key_version: "encryptionkey@0"
I20260929 20:29:19.660985 1368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:19.686426 1368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:19.692605 1368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:19.693200 1368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:19.694425 1368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:29:19.696334 1368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:29:19.696476 1368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:19.696622 1368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:29:19.696714 1368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:19.739203 1368 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.129:38899
I20260929 20:29:19.739306 1494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.129:38899 every 8 connection(s)
I20260929 20:29:19.740692 1368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
I20260929 20:29:19.750365 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 1368
I20260929 20:29:19.750625 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance
I20260929 20:29:19.751317 29650 external_mini_cluster.cc:1428] Setting key 53a086f3ec3481219f3806b80797cfdd
Sep 29 20:29:19 dist-test-slave-p29l krb5kdc[1263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713759, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.244.130@KRBTEST.COM; defaulting to no policy
I20260929 20:29:19.774170 1497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:19.743688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:60661 (local address 127.28.244.190:42573)
0929 20:29:19.746818 (+ 3130us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:19.746823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:19.749102 (+ 2279us) server_negotiation.cc:407] Connection header received
0929 20:29:19.751858 (+ 2756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:19.751866 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:19.752023 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:19.752158 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:19.754276 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:19.755698 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:19.757675 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:19.758568 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:19.758768 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:19.767568 (+ 8800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:19.767574 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:19.767577 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:19.767619 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:19.770487 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:19.771441 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:19.771450 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:19.771456 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:19.771600 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:19.772170 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:19.772178 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:19.772186 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:19.772529 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:19.772699 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:19.773524 (+ 825us) server_negotiation.cc:299] Negotiation successful
0929 20:29:19.773776 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2983,"thread_start_us":2888,"threads_started":1}
I20260929 20:29:19.775297 1498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:19.744709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:42573 (local address 127.28.244.129:60661)
0929 20:29:19.748867 (+ 4158us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:19.748928 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:19.751144 (+ 2216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:19.752624 (+ 1480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:19.752634 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:19.753208 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:19.754042 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:19.754064 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:19.755991 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:19.755996 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:19.757461 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:19.757478 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:19.758907 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:19.758914 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:19.759155 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:19.763587 (+ 4432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:19.763640 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:19.767242 (+ 3602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:19.770730 (+ 3488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:19.770742 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:19.770769 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:19.771289 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:19.771736 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:19.771741 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:19.771745 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:19.772040 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:19.772683 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:19.772692 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:19.773104 (+ 412us) client_negotiation.cc:769] Sending connection context
0929 20:29:19.773425 (+ 321us) client_negotiation.cc:240] Negotiation successful
0929 20:29:19.774034 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3804,"thread_start_us":3494,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.130@KRBTEST.COM" created.
I20260929 20:29:19.777258 1495 heartbeater.cc:347] Connected to a master server at 127.28.244.190:42573
I20260929 20:29:19.777654 1495 heartbeater.cc:464] Registering TS with master...
I20260929 20:29:19.778621 1495 heartbeater.cc:511] Master 127.28.244.190:42573 requested a full tablet report, sending...
I20260929 20:29:19.781538 1307 ts_manager.cc:194] Registered new tserver with Master: ec38e974d7114bb7aa05c1bd69a221a2 (127.28.244.129:38899)
I20260929 20:29:19.784242 1307 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.129@KRBTEST.COM'} at 127.28.244.129:60661
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab.
Entry for principal kudu/127.28.244.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab.
I20260929 20:29:19.805078 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.130:0
--local_ip_for_outbound_sockets=127.28.244.130
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:42573
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:44529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:29:20.195161 1501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:20.195578 1501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:20.195773 1501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:20.203967 1501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:20.204100 1501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:20.204222 1501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.130
I20260929 20:29:20.215240 1501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:44529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:42573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.130
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:20.216773 1501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:20.218233 1501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:20.234192 1506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:20.234220 1509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:20.235442 1507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:20.235479 1501 server_base.cc:1034] running on GCE node
I20260929 20:29:20.236358 1501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:20.237582 1501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:20.238868 1501 hybrid_clock.cc:648] HybridClock initialized: now 1790713760238816 us; error 42 us; skew 500 ppm
I20260929 20:29:20.239288 1501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:20 dist-test-slave-p29l krb5kdc[1263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713760, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:20.243126 1501 init.cc:377] Logged in from keytab as kudu/127.28.244.130@KRBTEST.COM (short username kudu)
I20260929 20:29:20.244586 1501 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:20.244705 1501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:20.245023 1501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:20.248600 1501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-1/data/instance:
uuid: "43409bf85793489aa896d28b18eb465f"
format_stamp: "Formatted at 2026-09-29 20:29:20 on dist-test-slave-p29l"
server_key: "bc262b5b5dcb9f9d93f811310f058ef9"
server_key_iv: "725068e5682f7ec6c7839d0434ce13f9"
server_key_version: "encryptionkey@0"
I20260929 20:29:20.249554 1501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance:
uuid: "43409bf85793489aa896d28b18eb465f"
format_stamp: "Formatted at 2026-09-29 20:29:20 on dist-test-slave-p29l"
server_key: "bc262b5b5dcb9f9d93f811310f058ef9"
server_key_iv: "725068e5682f7ec6c7839d0434ce13f9"
server_key_version: "encryptionkey@0"
I20260929 20:29:20.255929 1501 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 20:29:20.260520 1515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:20.262238 1501 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 20:29:20.262539 1501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-1/wal
uuid: "43409bf85793489aa896d28b18eb465f"
format_stamp: "Formatted at 2026-09-29 20:29:20 on dist-test-slave-p29l"
server_key: "bc262b5b5dcb9f9d93f811310f058ef9"
server_key_iv: "725068e5682f7ec6c7839d0434ce13f9"
server_key_version: "encryptionkey@0"
I20260929 20:29:20.262830 1501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:20.287133 1501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:20.291939 1501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:20.292245 1501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:20.293354 1501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:29:20.294871 1501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:29:20.294986 1501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:20.295156 1501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:29:20.295260 1501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:20.332136 1501 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.130:33309
I20260929 20:29:20.332191 1627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.130:33309 every 8 connection(s)
I20260929 20:29:20.333133 1501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
I20260929 20:29:20.339020 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 1501
I20260929 20:29:20.339242 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance
I20260929 20:29:20.339699 29650 external_mini_cluster.cc:1428] Setting key 960c017177e1b5b7b9d23b1b252fa4d3
Sep 29 20:29:20 dist-test-slave-p29l krb5kdc[1263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713760, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:29:20.357559 1630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:20.335900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:44175 (local address 127.28.244.190:42573)
0929 20:29:20.336406 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:20.336412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:20.337418 (+ 1006us) server_negotiation.cc:407] Connection header received
0929 20:29:20.339562 (+ 2144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:20.339568 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:20.339673 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:20.339842 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:20.341776 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:20.342505 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:20.344453 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:20.344909 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:20.345061 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:20.349972 (+ 4911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:20.349978 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:20.349982 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:20.350025 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:20.353133 (+ 3108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:20.354468 (+ 1335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:20.354474 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:20.354477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:20.354635 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:20.355259 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:20.355263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:20.355265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:20.355577 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:20.355704 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:20.357006 (+ 1302us) server_negotiation.cc:299] Negotiation successful
0929 20:29:20.357250 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":277,"threads_started":1}
I20260929 20:29:20.359433 1631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:20.336447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:42573 (local address 127.28.244.130:44175)
0929 20:29:20.337208 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:20.337259 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:20.339155 (+ 1896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:20.340115 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:20.340124 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:20.340844 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:20.341564 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:20.341594 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:20.342694 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:20.342704 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:20.344243 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:20.344260 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:20.345177 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:20.345184 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:20.345449 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:20.346359 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:20.346389 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:20.349575 (+ 3186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:20.353361 (+ 3786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:20.353377 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:20.353408 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:20.354258 (+ 850us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:20.354797 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:20.354806 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:20.354813 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:20.355118 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:20.355747 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:20.355761 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:20.356424 (+ 663us) client_negotiation.cc:769] Sending connection context
0929 20:29:20.356810 (+ 386us) client_negotiation.cc:240] Negotiation successful
0929 20:29:20.357575 (+ 765us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":318,"threads_started":1}
I20260929 20:29:20.361361 1628 heartbeater.cc:347] Connected to a master server at 127.28.244.190:42573
I20260929 20:29:20.361771 1628 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.28.244.131@KRBTEST.COM; defaulting to no policy
I20260929 20:29:20.362670 1628 heartbeater.cc:511] Master 127.28.244.190:42573 requested a full tablet report, sending...
I20260929 20:29:20.364480 1307 ts_manager.cc:194] Registered new tserver with Master: 43409bf85793489aa896d28b18eb465f (127.28.244.130:33309)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.131@KRBTEST.COM" created.
I20260929 20:29:20.365552 1307 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.130@KRBTEST.COM'} at 127.28.244.130:44175
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab.
Entry for principal kudu/127.28.244.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab.
I20260929 20:29:20.397056 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.131:0
--local_ip_for_outbound_sockets=127.28.244.131
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:42573
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:44529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:29:20.762221 1634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:20.762574 1634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:20.762768 1634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:20.771032 1634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:20.771173 1634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:20.771368 1634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.131
I20260929 20:29:20.781908 1634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:44529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:42573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.131
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:20.783360 1634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:20.785012 1634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 20:29:20.787865 1495 heartbeater.cc:503] Master 127.28.244.190:42573 was elected leader, sending a full tablet report...
W20260929 20:29:20.799995 1639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:20.800014 1642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:20.800074 1640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:20.800962 1634 server_base.cc:1034] running on GCE node
I20260929 20:29:20.801698 1634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:20.802865 1634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:20.804106 1634 hybrid_clock.cc:648] HybridClock initialized: now 1790713760804044 us; error 57 us; skew 500 ppm
I20260929 20:29:20.804566 1634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:20 dist-test-slave-p29l krb5kdc[1263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713760, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:20.808449 1634 init.cc:377] Logged in from keytab as kudu/127.28.244.131@KRBTEST.COM (short username kudu)
I20260929 20:29:20.809852 1634 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:20.809958 1634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:20.810353 1634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:20.813809 1634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-2/data/instance:
uuid: "9644f3438df2423e8752ccdf3883bb34"
format_stamp: "Formatted at 2026-09-29 20:29:20 on dist-test-slave-p29l"
server_key: "e6ad015706b6097a4b617d2faa866a88"
server_key_iv: "435bc740f51354e213aae783c8065742"
server_key_version: "encryptionkey@0"
I20260929 20:29:20.814761 1634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance:
uuid: "9644f3438df2423e8752ccdf3883bb34"
format_stamp: "Formatted at 2026-09-29 20:29:20 on dist-test-slave-p29l"
server_key: "e6ad015706b6097a4b617d2faa866a88"
server_key_iv: "435bc740f51354e213aae783c8065742"
server_key_version: "encryptionkey@0"
I20260929 20:29:20.820974 1634 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260929 20:29:20.825531 1648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:20.827262 1634 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 20:29:20.827535 1634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-2/wal
uuid: "9644f3438df2423e8752ccdf3883bb34"
format_stamp: "Formatted at 2026-09-29 20:29:20 on dist-test-slave-p29l"
server_key: "e6ad015706b6097a4b617d2faa866a88"
server_key_iv: "435bc740f51354e213aae783c8065742"
server_key_version: "encryptionkey@0"
I20260929 20:29:20.827874 1634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:20.860193 1634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:20.864989 1634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:20.865341 1634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:20.866638 1634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:29:20.868556 1634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:29:20.868685 1634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:20.868817 1634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:29:20.868913 1634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:20.905282 1634 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.131:41119
I20260929 20:29:20.905318 1760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.131:41119 every 8 connection(s)
I20260929 20:29:20.906370 1634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
I20260929 20:29:20.910900 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 1634
I20260929 20:29:20.911135 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance
I20260929 20:29:20.911724 29650 external_mini_cluster.cc:1428] Setting key cc872b7d2c9c2350614b570580ac40a2
Sep 29 20:29:20 dist-test-slave-p29l krb5kdc[1263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713760, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:29:20.929987 1763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:20.909083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:37605 (local address 127.28.244.190:42573)
0929 20:29:20.909613 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:20.909617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:20.910907 (+ 1290us) server_negotiation.cc:407] Connection header received
0929 20:29:20.913484 (+ 2577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:20.913489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:20.913592 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:20.913723 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:20.915805 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:20.916690 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:20.918448 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:20.918921 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:20.919074 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:20.923115 (+ 4041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:20.923121 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:20.923124 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:20.923163 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:20.925932 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:20.926896 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:20.926901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:20.926904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:20.927097 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:20.927699 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:20.927707 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:20.927711 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:20.928225 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:20.928413 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:20.929469 (+ 1056us) server_negotiation.cc:299] Negotiation successful
0929 20:29:20.929701 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":285,"threads_started":1}
I20260929 20:29:20.931044 1764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:20.909675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:42573 (local address 127.28.244.131:37605)
0929 20:29:20.910658 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:20.910717 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:20.913115 (+ 2398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:20.914007 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:20.914018 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:20.914812 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:20.915602 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:20.915624 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:20.916880 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:20.916885 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:20.918250 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:20.918265 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:20.919180 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:20.919187 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:20.919447 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:20.920242 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:20.920273 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:20.922826 (+ 2553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:20.926153 (+ 3327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:20.926164 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:20.926195 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:20.926755 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:20.927261 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:20.927267 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:20.927270 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:20.927538 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:20.928448 (+ 910us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:20.928457 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:20.929023 (+ 566us) client_negotiation.cc:769] Sending connection context
0929 20:29:20.929316 (+ 293us) client_negotiation.cc:240] Negotiation successful
0929 20:29:20.929773 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":355,"threads_started":1}
I20260929 20:29:20.932862 1761 heartbeater.cc:347] Connected to a master server at 127.28.244.190:42573
I20260929 20:29:20.933226 1761 heartbeater.cc:464] Registering TS with master...
I20260929 20:29:20.933991 1761 heartbeater.cc:511] Master 127.28.244.190:42573 requested a full tablet report, sending...
I20260929 20:29:20.935694 1307 ts_manager.cc:194] Registered new tserver with Master: 9644f3438df2423e8752ccdf3883bb34 (127.28.244.131:41119)
I20260929 20:29:20.936592 1307 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.131@KRBTEST.COM'} at 127.28.244.131:37605
I20260929 20:29:20.942013 29650 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:29:20.961263 1763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:20.947026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57154 (local address 127.28.244.190:42573)
0929 20:29:20.947216 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:20.947221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:20.947897 (+ 676us) server_negotiation.cc:407] Connection header received
0929 20:29:20.948113 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:20.948118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:20.948202 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:20.948309 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:20.949932 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:20.950724 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:20.952177 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:20.952663 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:20.952819 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:20.955324 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:20.955330 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:20.955333 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:20.955374 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:20.958217 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:20.959038 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:20.959044 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:20.959047 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:20.959193 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:20.959589 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:20.959594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:20.959597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:20.960038 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:20.960310 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:20.960658 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 20:29:20.960924 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:29:20.966260 1307 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:29:20.974319 1307 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:29:21.015981 1776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:20.989216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58278 (local address 127.28.244.130:33309)
0929 20:29:21.006172 (+ 16956us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:21.006185 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:21.006230 (+ 45us) server_negotiation.cc:407] Connection header received
0929 20:29:21.006368 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:21.006381 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:21.006759 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:21.007026 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:29:21.008507 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:21.010149 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:21.013099 (+ 2950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:21.014545 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:21.014750 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:21.014974 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:21.015377 (+ 403us) server_negotiation.cc:299] Negotiation successful
0929 20:29:21.015576 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16690,"thread_start_us":17482,"threads_started":1}
I20260929 20:29:21.016184 1777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:20.988759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:33309 (local address 127.0.0.1:58278)
0929 20:29:21.004973 (+ 16214us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:21.005035 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:21.005394 (+ 359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:21.007485 (+ 2091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:21.007498 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:21.007548 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:29:21.008217 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:21.008233 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:21.010315 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:21.010320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:21.012834 (+ 2514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:21.012873 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:21.014775 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:21.014781 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:21.015112 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:21.015161 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 20:29:21.015304 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 20:29:21.015550 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15862,"mutex_wait_us":52,"thread_start_us":19447,"threads_started":1}
I20260929 20:29:21.017686 1774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:20.988754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:41119 (local address 127.0.0.1:50350)
0929 20:29:21.008482 (+ 19728us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:21.008504 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:21.008648 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:21.009555 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:21.009559 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:21.009591 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:29:21.010091 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:21.010104 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:21.011630 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:21.011634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:21.014036 (+ 2402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:21.014053 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:21.016733 (+ 2680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:21.016740 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:21.016943 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:21.016983 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 20:29:21.017101 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 20:29:21.017263 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19603,"mutex_wait_us":706,"thread_start_us":15058,"threads_started":1}
I20260929 20:29:21.017803 1778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:20.989284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50350 (local address 127.28.244.131:41119)
0929 20:29:21.004159 (+ 14875us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:21.004173 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:21.008652 (+ 4479us) server_negotiation.cc:407] Connection header received
0929 20:29:21.008778 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:21.008785 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:21.009044 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:21.009251 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:29:21.010330 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:21.011437 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:21.014271 (+ 2834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:21.015980 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:21.016187 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:21.016377 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:21.017207 (+ 830us) server_negotiation.cc:299] Negotiation successful
0929 20:29:21.017411 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14566,"thread_start_us":14367,"threads_started":1}
I20260929 20:29:21.019500 1775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:20.989101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:38899 (local address 127.0.0.1:57360)
0929 20:29:21.008135 (+ 19034us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:21.008164 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:21.008340 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:21.009380 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:21.009385 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:21.009413 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:29:21.010001 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:21.010017 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:21.012121 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:21.012126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:21.015588 (+ 3462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:21.015606 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:21.018631 (+ 3025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:21.018638 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:21.018824 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:21.018859 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 20:29:21.018954 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 20:29:21.019104 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18862,"mutex_wait_us":148,"thread_start_us":18704,"threads_started":1}
I20260929 20:29:21.019867 1779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:20.989745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57360 (local address 127.28.244.129:38899)
0929 20:29:21.007525 (+ 17780us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:21.007536 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:21.008481 (+ 945us) server_negotiation.cc:407] Connection header received
0929 20:29:21.008589 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:21.008599 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:21.008909 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:21.009135 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:29:21.010227 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:21.011953 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:21.015862 (+ 3909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:21.018018 (+ 2156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:21.018262 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:21.018460 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:21.019239 (+ 779us) server_negotiation.cc:299] Negotiation successful
0929 20:29:21.019452 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17568,"thread_start_us":17401,"threads_started":1}
I20260929 20:29:21.020982 1563 tablet_service.cc:1467] Processing CreateTablet for tablet a5eda6ae572541d7bb13c9386ceca325 (DEFAULT_TABLE table=test-table [id=8f20385de7eb40a6b3f153fc4b67c3dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:29:21.021958 1696 tablet_service.cc:1467] Processing CreateTablet for tablet a5eda6ae572541d7bb13c9386ceca325 (DEFAULT_TABLE table=test-table [id=8f20385de7eb40a6b3f153fc4b67c3dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:29:21.022734 1563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5eda6ae572541d7bb13c9386ceca325. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:21.023442 1696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5eda6ae572541d7bb13c9386ceca325. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:21.023514 1430 tablet_service.cc:1467] Processing CreateTablet for tablet a5eda6ae572541d7bb13c9386ceca325 (DEFAULT_TABLE table=test-table [id=8f20385de7eb40a6b3f153fc4b67c3dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:29:21.025272 1430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5eda6ae572541d7bb13c9386ceca325. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:21.040751 1781 tablet_bootstrap.cc:492] T a5eda6ae572541d7bb13c9386ceca325 P 43409bf85793489aa896d28b18eb465f: Bootstrap starting.
I20260929 20:29:21.043788 1780 tablet_bootstrap.cc:492] T a5eda6ae572541d7bb13c9386ceca325 P ec38e974d7114bb7aa05c1bd69a221a2: Bootstrap starting.
I20260929 20:29:21.045392 1782 tablet_bootstrap.cc:492] T a5eda6ae572541d7bb13c9386ceca325 P 9644f3438df2423e8752ccdf3883bb34: Bootstrap starting.
I20260929 20:29:21.045691 1781 tablet_bootstrap.cc:654] T a5eda6ae572541d7bb13c9386ceca325 P 43409bf85793489aa896d28b18eb465f: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:21.047219 1781 log.cc:824] T a5eda6ae572541d7bb13c9386ceca325 P 43409bf85793489aa896d28b18eb465f: Log is configured to *not* fsync() on all Append() calls
I20260929 20:29:21.047917 1780 tablet_bootstrap.cc:654] T a5eda6ae572541d7bb13c9386ceca325 P ec38e974d7114bb7aa05c1bd69a221a2: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:21.049582 1780 log.cc:824] T a5eda6ae572541d7bb13c9386ceca325 P ec38e974d7114bb7aa05c1bd69a221a2: Log is configured to *not* fsync() on all Append() calls
I20260929 20:29:21.049935 1782 tablet_bootstrap.cc:654] T a5eda6ae572541d7bb13c9386ceca325 P 9644f3438df2423e8752ccdf3883bb34: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:21.051890 1782 log.cc:824] T a5eda6ae572541d7bb13c9386ceca325 P 9644f3438df2423e8752ccdf3883bb34: Log is configured to *not* fsync() on all Append() calls
I20260929 20:29:21.057467 1781 tablet_bootstrap.cc:492] T a5eda6ae572541d7bb13c9386ceca325 P 43409bf85793489aa896d28b18eb465f: No bootstrap required, opened a new log
I20260929 20:29:21.057467 1782 tablet_bootstrap.cc:492] T a5eda6ae572541d7bb13c9386ceca325 P 9644f3438df2423e8752ccdf3883bb34: No bootstrap required, opened a new log
I20260929 20:29:21.057819 1782 ts_tablet_manager.cc:1397] T a5eda6ae572541d7bb13c9386ceca325 P 9644f3438df2423e8752ccdf3883bb34: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260929 20:29:21.057852 1781 ts_tablet_manager.cc:1397] T a5eda6ae572541d7bb13c9386ceca325 P 43409bf85793489aa896d28b18eb465f: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.007s
I20260929 20:29:21.059362 1780 tablet_bootstrap.cc:492] T a5eda6ae572541d7bb13c9386ceca325 P ec38e974d7114bb7aa05c1bd69a221a2: No bootstrap required, opened a new log
I20260929 20:29:21.059799 1780 ts_tablet_manager.cc:1397] T a5eda6ae572541d7bb13c9386ceca325 P ec38e974d7114bb7aa05c1bd69a221a2: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.011s
I20260929 20:29:21.070118 1782 raft_consensus.cc:348] T a5eda6ae572541d7bb13c9386ceca325 P 9644f3438df2423e8752ccdf3883bb34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43409bf85793489aa896d28b18eb465f" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33309 } } peers { permanent_uuid: "9644f3438df2423e8752ccdf3883bb34" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 41119 } } peers { permanent_uuid: "ec38e974d7114bb7aa05c1bd69a221a2" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38899 } }
I20260929 20:29:21.070911 1782 raft_consensus.cc:374] T a5eda6ae572541d7bb13c9386ceca325 P 9644f3438df2423e8752ccdf3883bb34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:21.071091 1782 raft_consensus.cc:729] T a5eda6ae572541d7bb13c9386ceca325 P 9644f3438df2423e8752ccdf3883bb34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9644f3438df2423e8752ccdf3883bb34, State: Initialized, Role: FOLLOWER
I20260929 20:29:21.070924 1781 raft_consensus.cc:348] T a5eda6ae572541d7bb13c9386ceca325 P 43409bf85793489aa896d28b18eb465f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43409bf85793489aa896d28b18eb465f" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33309 } } peers { permanent_uuid: "9644f3438df2423e8752ccdf3883bb34" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 41119 } } peers { permanent_uuid: "ec38e974d7114bb7aa05c1bd69a221a2" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38899 } }
I20260929 20:29:21.071300 1780 raft_consensus.cc:348] T a5eda6ae572541d7bb13c9386ceca325 P ec38e974d7114bb7aa05c1bd69a221a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43409bf85793489aa896d28b18eb465f" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33309 } } peers { permanent_uuid: "9644f3438df2423e8752ccdf3883bb34" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 41119 } } peers { permanent_uuid: "ec38e974d7114bb7aa05c1bd69a221a2" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38899 } }
I20260929 20:29:21.071463 1781 raft_consensus.cc:374] T a5eda6ae572541d7bb13c9386ceca325 P 43409bf85793489aa896d28b18eb465f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:21.071702 1781 raft_consensus.cc:729] T a5eda6ae572541d7bb13c9386ceca325 P 43409bf85793489aa896d28b18eb465f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43409bf85793489aa896d28b18eb465f, State: Initialized, Role: FOLLOWER
I20260929 20:29:21.071832 1780 raft_consensus.cc:374] T a5eda6ae572541d7bb13c9386ceca325 P ec38e974d7114bb7aa05c1bd69a221a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:21.072010 1780 raft_consensus.cc:729] T a5eda6ae572541d7bb13c9386ceca325 P ec38e974d7114bb7aa05c1bd69a221a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec38e974d7114bb7aa05c1bd69a221a2, State: Initialized, Role: FOLLOWER
I20260929 20:29:21.071964 1782 consensus_queue.cc:260] T a5eda6ae572541d7bb13c9386ceca325 P 9644f3438df2423e8752ccdf3883bb34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43409bf85793489aa896d28b18eb465f" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33309 } } peers { permanent_uuid: "9644f3438df2423e8752ccdf3883bb34" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 41119 } } peers { permanent_uuid: "ec38e974d7114bb7aa05c1bd69a221a2" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38899 } }
I20260929 20:29:21.072556 1781 consensus_queue.cc:260] T a5eda6ae572541d7bb13c9386ceca325 P 43409bf85793489aa896d28b18eb465f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43409bf85793489aa896d28b18eb465f" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33309 } } peers { permanent_uuid: "9644f3438df2423e8752ccdf3883bb34" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 41119 } } peers { permanent_uuid: "ec38e974d7114bb7aa05c1bd69a221a2" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38899 } }
I20260929 20:29:21.072804 1780 consensus_queue.cc:260] T a5eda6ae572541d7bb13c9386ceca325 P ec38e974d7114bb7aa05c1bd69a221a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43409bf85793489aa896d28b18eb465f" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33309 } } peers { permanent_uuid: "9644f3438df2423e8752ccdf3883bb34" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 41119 } } peers { permanent_uuid: "ec38e974d7114bb7aa05c1bd69a221a2" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38899 } }
I20260929 20:29:21.081055 1780 ts_tablet_manager.cc:1428] T a5eda6ae572541d7bb13c9386ceca325 P ec38e974d7114bb7aa05c1bd69a221a2: Time spent starting tablet: real 0.021s user 0.015s sys 0.004s
I20260929 20:29:21.081624 1781 ts_tablet_manager.cc:1428] T a5eda6ae572541d7bb13c9386ceca325 P 43409bf85793489aa896d28b18eb465f: Time spent starting tablet: real 0.024s user 0.010s sys 0.008s
I20260929 20:29:21.082131 1628 heartbeater.cc:503] Master 127.28.244.190:42573 was elected leader, sending a full tablet report...
I20260929 20:29:21.082571 1789 raft_consensus.cc:482] T a5eda6ae572541d7bb13c9386ceca325 P 9644f3438df2423e8752ccdf3883bb34 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:29:21.083082 1761 heartbeater.cc:503] Master 127.28.244.190:42573 was elected leader, sending a full tablet report...
I20260929 20:29:21.082880 1789 raft_consensus.cc:504] T a5eda6ae572541d7bb13c9386ceca325 P 9644f3438df2423e8752ccdf3883bb34 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43409bf85793489aa896d28b18eb465f" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33309 } } peers { permanent_uuid: "9644f3438df2423e8752ccdf3883bb34" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 41119 } } peers { permanent_uuid: "ec38e974d7114bb7aa05c1bd69a221a2" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38899 } }
W20260929 20:29:21.085130 1629 tablet.cc:2369] T a5eda6ae572541d7bb13c9386ceca325 P 43409bf85793489aa896d28b18eb465f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:29:21.086061 1782 ts_tablet_manager.cc:1428] T a5eda6ae572541d7bb13c9386ceca325 P 9644f3438df2423e8752ccdf3883bb34: Time spent starting tablet: real 0.028s user 0.018s sys 0.007s
I20260929 20:29:21.086412 1789 leader_election.cc:288] T a5eda6ae572541d7bb13c9386ceca325 P 9644f3438df2423e8752ccdf3883bb34 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43409bf85793489aa896d28b18eb465f (127.28.244.130:33309), ec38e974d7114bb7aa05c1bd69a221a2 (127.28.244.129:38899)
I20260929 20:29:21.094985 1764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:21.086648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:33309 (local address 127.28.244.131:37625)
0929 20:29:21.086995 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:21.087017 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:21.087136 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:21.087489 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:21.087494 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:21.087516 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:29:21.087709 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:21.087719 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:21.090542 (+ 2823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:21.090549 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:21.092639 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:21.092652 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:21.094232 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:21.094238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:21.094493 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:21.094533 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 20:29:21.094630 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 20:29:21.094740 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":1}
I20260929 20:29:21.095144 1776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:21.086906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:37625 (local address 127.28.244.130:33309)
0929 20:29:21.087161 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:21.087168 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:21.087190 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:29:21.087255 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:21.087260 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:21.087362 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:21.087471 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:29:21.088785 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:21.090301 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:21.092838 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:21.094058 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:21.094257 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:21.094339 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:21.094687 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 20:29:21.094887 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260929 20:29:21.096225 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5eda6ae572541d7bb13c9386ceca325" candidate_uuid: "9644f3438df2423e8752ccdf3883bb34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43409bf85793489aa896d28b18eb465f" is_pre_election: true
I20260929 20:29:21.096951 1583 raft_consensus.cc:2454] T a5eda6ae572541d7bb13c9386ceca325 P 43409bf85793489aa896d28b18eb465f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9644f3438df2423e8752ccdf3883bb34 in term 0.
I20260929 20:29:21.097540 1779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:21.087105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:42413 (local address 127.28.244.129:38899)
0929 20:29:21.087917 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:21.087924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:21.090924 (+ 3000us) server_negotiation.cc:407] Connection header received
0929 20:29:21.091134 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:21.091144 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:21.091206 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:21.091303 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:29:21.092004 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:21.093024 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:21.095449 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:21.096360 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:21.096470 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:21.096524 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:21.097101 (+ 577us) server_negotiation.cc:299] Negotiation successful
0929 20:29:21.097258 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578}
I20260929 20:29:21.097590 1790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:21.086649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:38899 (local address 127.28.244.131:42413)
0929 20:29:21.090801 (+ 4152us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:21.090821 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:21.090974 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:21.091320 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:21.091325 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:21.091353 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:29:21.091740 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:21.091749 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:21.093184 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:21.093189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:21.095207 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:21.095226 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:21.096717 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:21.096722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:21.096916 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:21.096941 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 20:29:21.097020 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 20:29:21.097126 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4003,"mutex_wait_us":568,"thread_start_us":3214,"threads_started":1}
I20260929 20:29:21.097870 1649 leader_election.cc:302] T a5eda6ae572541d7bb13c9386ceca325 P 9644f3438df2423e8752ccdf3883bb34 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43409bf85793489aa896d28b18eb465f, 9644f3438df2423e8752ccdf3883bb34; no voters:
I20260929 20:29:21.098390 1789 raft_consensus.cc:2781] T a5eda6ae572541d7bb13c9386ceca325 P 9644f3438df2423e8752ccdf3883bb34 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:29:21.098542 1789 raft_consensus.cc:482] T a5eda6ae572541d7bb13c9386ceca325 P 9644f3438df2423e8752ccdf3883bb34 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:29:21.098663 1789 raft_consensus.cc:3037] T a5eda6ae572541d7bb13c9386ceca325 P 9644f3438df2423e8752ccdf3883bb34 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:21.098627 1450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5eda6ae572541d7bb13c9386ceca325" candidate_uuid: "9644f3438df2423e8752ccdf3883bb34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec38e974d7114bb7aa05c1bd69a221a2" is_pre_election: true
I20260929 20:29:21.099216 1450 raft_consensus.cc:2454] T a5eda6ae572541d7bb13c9386ceca325 P ec38e974d7114bb7aa05c1bd69a221a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9644f3438df2423e8752ccdf3883bb34 in term 0.
I20260929 20:29:21.101228 1789 raft_consensus.cc:504] T a5eda6ae572541d7bb13c9386ceca325 P 9644f3438df2423e8752ccdf3883bb34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43409bf85793489aa896d28b18eb465f" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33309 } } peers { permanent_uuid: "9644f3438df2423e8752ccdf3883bb34" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 41119 } } peers { permanent_uuid: "ec38e974d7114bb7aa05c1bd69a221a2" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38899 } }
I20260929 20:29:21.101890 1789 leader_election.cc:288] T a5eda6ae572541d7bb13c9386ceca325 P 9644f3438df2423e8752ccdf3883bb34 [CANDIDATE]: Term 1 election: Requested vote from peers 43409bf85793489aa896d28b18eb465f (127.28.244.130:33309), ec38e974d7114bb7aa05c1bd69a221a2 (127.28.244.129:38899)
I20260929 20:29:21.102561 1450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5eda6ae572541d7bb13c9386ceca325" candidate_uuid: "9644f3438df2423e8752ccdf3883bb34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec38e974d7114bb7aa05c1bd69a221a2"
I20260929 20:29:21.102561 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5eda6ae572541d7bb13c9386ceca325" candidate_uuid: "9644f3438df2423e8752ccdf3883bb34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43409bf85793489aa896d28b18eb465f"
I20260929 20:29:21.102818 1583 raft_consensus.cc:3037] T a5eda6ae572541d7bb13c9386ceca325 P 43409bf85793489aa896d28b18eb465f [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:21.102818 1450 raft_consensus.cc:3037] T a5eda6ae572541d7bb13c9386ceca325 P ec38e974d7114bb7aa05c1bd69a221a2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:21.106348 1583 raft_consensus.cc:2454] T a5eda6ae572541d7bb13c9386ceca325 P 43409bf85793489aa896d28b18eb465f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9644f3438df2423e8752ccdf3883bb34 in term 1.
I20260929 20:29:21.106348 1450 raft_consensus.cc:2454] T a5eda6ae572541d7bb13c9386ceca325 P ec38e974d7114bb7aa05c1bd69a221a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9644f3438df2423e8752ccdf3883bb34 in term 1.
I20260929 20:29:21.106882 1649 leader_election.cc:302] T a5eda6ae572541d7bb13c9386ceca325 P 9644f3438df2423e8752ccdf3883bb34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43409bf85793489aa896d28b18eb465f, 9644f3438df2423e8752ccdf3883bb34; no voters:
I20260929 20:29:21.107231 1789 raft_consensus.cc:2781] T a5eda6ae572541d7bb13c9386ceca325 P 9644f3438df2423e8752ccdf3883bb34 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:29:21.107405 1789 raft_consensus.cc:686] T a5eda6ae572541d7bb13c9386ceca325 P 9644f3438df2423e8752ccdf3883bb34 [term 1 LEADER]: Becoming Leader. State: Replica: 9644f3438df2423e8752ccdf3883bb34, State: Running, Role: LEADER
I20260929 20:29:21.108016 1789 consensus_queue.cc:237] T a5eda6ae572541d7bb13c9386ceca325 P 9644f3438df2423e8752ccdf3883bb34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43409bf85793489aa896d28b18eb465f" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33309 } } peers { permanent_uuid: "9644f3438df2423e8752ccdf3883bb34" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 41119 } } peers { permanent_uuid: "ec38e974d7114bb7aa05c1bd69a221a2" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38899 } }
I20260929 20:29:21.113081 1306 catalog_manager.cc:5495] T a5eda6ae572541d7bb13c9386ceca325 P 9644f3438df2423e8752ccdf3883bb34 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9644f3438df2423e8752ccdf3883bb34 (127.28.244.131). New cstate: current_term: 1 leader_uuid: "9644f3438df2423e8752ccdf3883bb34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43409bf85793489aa896d28b18eb465f" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9644f3438df2423e8752ccdf3883bb34" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 41119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec38e974d7114bb7aa05c1bd69a221a2" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38899 } health_report { overall_health: UNKNOWN } } }
I20260929 20:29:21.149962 1778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:21.143509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50358 (local address 127.28.244.131:41119)
0929 20:29:21.143734 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:21.143739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:21.143760 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:29:21.144062 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:21.144067 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:21.144168 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:21.144361 (+ 193us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:29:21.144753 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:21.145712 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:21.147447 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:21.147993 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:21.148112 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:21.148719 (+ 607us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:29:21.149312 (+ 593us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:29:21.149443 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:21.149653 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 20:29:21.149744 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260929 20:29:21.158059 1762 tablet.cc:2369] T a5eda6ae572541d7bb13c9386ceca325 P 9644f3438df2423e8752ccdf3883bb34: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:29:21.159853 1450 raft_consensus.cc:1261] T a5eda6ae572541d7bb13c9386ceca325 P ec38e974d7114bb7aa05c1bd69a221a2 [term 1 FOLLOWER]: Refusing update from remote peer 9644f3438df2423e8752ccdf3883bb34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:29:21.159873 1583 raft_consensus.cc:1261] T a5eda6ae572541d7bb13c9386ceca325 P 43409bf85793489aa896d28b18eb465f [term 1 FOLLOWER]: Refusing update from remote peer 9644f3438df2423e8752ccdf3883bb34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:29:21.161053 1789 consensus_queue.cc:1035] T a5eda6ae572541d7bb13c9386ceca325 P 9644f3438df2423e8752ccdf3883bb34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43409bf85793489aa896d28b18eb465f" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:29:21.161408 1786 consensus_queue.cc:1035] T a5eda6ae572541d7bb13c9386ceca325 P 9644f3438df2423e8752ccdf3883bb34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec38e974d7114bb7aa05c1bd69a221a2" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:29:21.194309 1307 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:29:21.194674 1307 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:29:21.199265 1307 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 95843243b469455c9d092e0511efab91: Sending DeleteTablet for 3 replicas of tablet a5eda6ae572541d7bb13c9386ceca325
I20260929 20:29:21.200732 1696 tablet_service.cc:1514] Processing DeleteTablet for tablet a5eda6ae572541d7bb13c9386ceca325 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:29:21 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:50350
I20260929 20:29:21.200780 1563 tablet_service.cc:1514] Processing DeleteTablet for tablet a5eda6ae572541d7bb13c9386ceca325 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:29:21 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:58278
I20260929 20:29:21.200780 1430 tablet_service.cc:1514] Processing DeleteTablet for tablet a5eda6ae572541d7bb13c9386ceca325 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:29:21 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:57360
I20260929 20:29:21.202148 1803 tablet_replica.cc:330] T a5eda6ae572541d7bb13c9386ceca325 P 9644f3438df2423e8752ccdf3883bb34: stopping tablet replica
I20260929 20:29:21.202791 1803 raft_consensus.cc:2229] T a5eda6ae572541d7bb13c9386ceca325 P 9644f3438df2423e8752ccdf3883bb34 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:29:21.203085 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 1368
I20260929 20:29:21.203576 1803 raft_consensus.cc:2258] T a5eda6ae572541d7bb13c9386ceca325 P 9644f3438df2423e8752ccdf3883bb34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:29:21.204619 1805 tablet_replica.cc:330] T a5eda6ae572541d7bb13c9386ceca325 P 43409bf85793489aa896d28b18eb465f: stopping tablet replica
I20260929 20:29:21.205112 1805 raft_consensus.cc:2229] T a5eda6ae572541d7bb13c9386ceca325 P 43409bf85793489aa896d28b18eb465f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:29:21.205750 1805 raft_consensus.cc:2258] T a5eda6ae572541d7bb13c9386ceca325 P 43409bf85793489aa896d28b18eb465f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:29:21.206627 1804 tablet_replica.cc:330] T a5eda6ae572541d7bb13c9386ceca325 P ec38e974d7114bb7aa05c1bd69a221a2: stopping tablet replica
I20260929 20:29:21.207070 1804 raft_consensus.cc:2229] T a5eda6ae572541d7bb13c9386ceca325 P ec38e974d7114bb7aa05c1bd69a221a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:29:21.207605 1804 raft_consensus.cc:2258] T a5eda6ae572541d7bb13c9386ceca325 P ec38e974d7114bb7aa05c1bd69a221a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:29:21.208365 1803 ts_tablet_manager.cc:1905] T a5eda6ae572541d7bb13c9386ceca325 P 9644f3438df2423e8752ccdf3883bb34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:29:21.209950 1805 ts_tablet_manager.cc:1905] T a5eda6ae572541d7bb13c9386ceca325 P 43409bf85793489aa896d28b18eb465f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:29:21.217679 1803 ts_tablet_manager.cc:1918] T a5eda6ae572541d7bb13c9386ceca325 P 9644f3438df2423e8752ccdf3883bb34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:29:21.218068 1803 log.cc:1196] T a5eda6ae572541d7bb13c9386ceca325 P 9644f3438df2423e8752ccdf3883bb34: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-2/wal/wals/a5eda6ae572541d7bb13c9386ceca325
I20260929 20:29:21.218806 1803 ts_tablet_manager.cc:1939] T a5eda6ae572541d7bb13c9386ceca325 P 9644f3438df2423e8752ccdf3883bb34: Deleting consensus metadata
I20260929 20:29:21.221093 1294 catalog_manager.cc:4841] TS 9644f3438df2423e8752ccdf3883bb34 (127.28.244.131:41119): tablet a5eda6ae572541d7bb13c9386ceca325 (table test-table [id=8f20385de7eb40a6b3f153fc4b67c3dc]) successfully deleted
I20260929 20:29:21.221326 1805 ts_tablet_manager.cc:1918] T a5eda6ae572541d7bb13c9386ceca325 P 43409bf85793489aa896d28b18eb465f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:29:21.221520 1805 log.cc:1196] T a5eda6ae572541d7bb13c9386ceca325 P 43409bf85793489aa896d28b18eb465f: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-1/wal/wals/a5eda6ae572541d7bb13c9386ceca325
I20260929 20:29:21.222435 1804 ts_tablet_manager.cc:1905] T a5eda6ae572541d7bb13c9386ceca325 P ec38e974d7114bb7aa05c1bd69a221a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:29:21.222803 1805 ts_tablet_manager.cc:1939] T a5eda6ae572541d7bb13c9386ceca325 P 43409bf85793489aa896d28b18eb465f: Deleting consensus metadata
I20260929 20:29:21.225109 1291 catalog_manager.cc:4841] TS 43409bf85793489aa896d28b18eb465f (127.28.244.130:33309): tablet a5eda6ae572541d7bb13c9386ceca325 (table test-table [id=8f20385de7eb40a6b3f153fc4b67c3dc]) successfully deleted
Sep 29 20:29:21 dist-test-slave-p29l krb5kdc[1263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.129@KRBTEST.COM
I20260929 20:29:21.232015 1804 ts_tablet_manager.cc:1918] T a5eda6ae572541d7bb13c9386ceca325 P ec38e974d7114bb7aa05c1bd69a221a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:29:21.232317 1804 log.cc:1196] T a5eda6ae572541d7bb13c9386ceca325 P ec38e974d7114bb7aa05c1bd69a221a2: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713721201454-29650-0/minicluster-data/ts-0/wal/wals/a5eda6ae572541d7bb13c9386ceca325
I20260929 20:29:21.232946 1804 ts_tablet_manager.cc:1939] T a5eda6ae572541d7bb13c9386ceca325 P ec38e974d7114bb7aa05c1bd69a221a2: Deleting consensus metadata
I20260929 20:29:21.235219 1292 catalog_manager.cc:4841] TS ec38e974d7114bb7aa05c1bd69a221a2 (127.28.244.129:38899): tablet a5eda6ae572541d7bb13c9386ceca325 (table test-table [id=8f20385de7eb40a6b3f153fc4b67c3dc]) successfully deleted
I20260929 20:29:21.235931 1779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:21.206760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57368 (local address 127.28.244.129:38899)
0929 20:29:21.206951 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:21.206958 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:21.215330 (+ 8372us) server_negotiation.cc:407] Connection header received
0929 20:29:21.215765 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:21.215772 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:21.215911 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:21.216005 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:21.218296 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:21.219507 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:21.221522 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:21.222339 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:21.222529 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:21.228344 (+ 5815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:21.228360 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:21.228377 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:21.228432 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:21.232459 (+ 4027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:21.233503 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:21.233515 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:21.233518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:21.233703 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:21.234190 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:21.234196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:21.234199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:21.234651 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:21.235010 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:21.235343 (+ 333us) server_negotiation.cc:299] Negotiation successful
0929 20:29:21.235587 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 20:29:21.237224 1490 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:29:21.335901 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 1368
I20260929 20:29:21.360839 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 1501
Sep 29 20:29:21 dist-test-slave-p29l krb5kdc[1263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.130@KRBTEST.COM
I20260929 20:29:21.378104 1776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:21.361927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58280 (local address 127.28.244.130:33309)
0929 20:29:21.362135 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:21.362141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:21.362164 (+ 23us) server_negotiation.cc:407] Connection header received
0929 20:29:21.362337 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:21.362342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:21.362416 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:21.362540 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:21.363796 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:21.364642 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:21.365981 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:21.366417 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:21.366569 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:21.371579 (+ 5010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:21.371598 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:21.371618 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:21.371665 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:21.374756 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:21.375928 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:21.375936 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:21.375938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:21.376085 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:21.376550 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:21.376554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:21.376556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:21.377001 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:21.377242 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:21.377660 (+ 418us) server_negotiation.cc:299] Negotiation successful
0929 20:29:21.377849 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 20:29:21.378975 1623 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:29:21.473888 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 1501
I20260929 20:29:21.494086 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 1634
Sep 29 20:29:21 dist-test-slave-p29l krb5kdc[1263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.131@KRBTEST.COM
I20260929 20:29:21.515241 1778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:21.494981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50368 (local address 127.28.244.131:41119)
0929 20:29:21.495160 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:21.495165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:21.495378 (+ 213us) server_negotiation.cc:407] Connection header received
0929 20:29:21.495533 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:21.495538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:21.496003 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:21.496108 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:21.498102 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:21.498981 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:21.501116 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:21.501687 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:21.501826 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:21.507796 (+ 5970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:21.507869 (+ 73us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:21.507898 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:21.507958 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:21.511354 (+ 3396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:21.512638 (+ 1284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:21.512647 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:21.512650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:21.512881 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:21.513351 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:21.513355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:21.513357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:21.513788 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:21.514255 (+ 467us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:21.514551 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 20:29:21.514813 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 20:29:21.516047 1756 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:29:21.645411 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 1634
I20260929 20:29:21.662624 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 1276
I20260929 20:29:21.664080 1337 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:29:21.779723 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 1276
2026-09-29T20:29:21Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4625 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T20:29:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:29:21Z Disabled control of system clock
I20260929 20:29:21.853319 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:33071
--webserver_interface=127.28.244.190
--webserver_port=0
--builtin_ntp_servers=127.28.244.148:45555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:33071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 20:29:22.211405 1814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:22.211740 1814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:22.211849 1814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:22.219636 1814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:29:22.219771 1814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:22.219880 1814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:22.219969 1814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:29:22.220018 1814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:29:22.230094 1814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:33071
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:33071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:22.231474 1814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:22.233552 1814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:22.246757 1820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:22.246881 1822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:22.246855 1819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:22.247745 1814 server_base.cc:1034] running on GCE node
I20260929 20:29:22.248736 1814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:22.250516 1814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:22.251888 1814 hybrid_clock.cc:648] HybridClock initialized: now 1790713762251767 us; error 73 us; skew 500 ppm
I20260929 20:29:22.252348 1814 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:29:22.255136 1814 webserver.cc:460] Webserver started at http://127.28.244.190:37087/ using document root <none> and password file <none>
I20260929 20:29:22.256003 1814 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:22.256145 1814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:22.256506 1814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:22.260098 1814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "5528391355604b43a374cb7c078210d8"
format_stamp: "Formatted at 2026-09-29 20:29:22 on dist-test-slave-p29l"
server_key: "1124b0620c4aad1645283fc3243dee06"
server_key_iv: "6c2293849649f98a7f8b644eb394b5f6"
server_key_version: "encryptionkey@0"
I20260929 20:29:22.261087 1814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "5528391355604b43a374cb7c078210d8"
format_stamp: "Formatted at 2026-09-29 20:29:22 on dist-test-slave-p29l"
server_key: "1124b0620c4aad1645283fc3243dee06"
server_key_iv: "6c2293849649f98a7f8b644eb394b5f6"
server_key_version: "encryptionkey@0"
I20260929 20:29:22.267190 1814 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260929 20:29:22.271322 1828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:22.272984 1814 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:29:22.273208 1814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "5528391355604b43a374cb7c078210d8"
format_stamp: "Formatted at 2026-09-29 20:29:22 on dist-test-slave-p29l"
server_key: "1124b0620c4aad1645283fc3243dee06"
server_key_iv: "6c2293849649f98a7f8b644eb394b5f6"
server_key_version: "encryptionkey@0"
I20260929 20:29:22.273528 1814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:22.280321 1814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:22.281262 1814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:22.281646 1814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:22.301443 1814 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:33071
I20260929 20:29:22.301448 1879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:33071 every 8 connection(s)
I20260929 20:29:22.302973 1814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:29:22.306015 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 1814
I20260929 20:29:22.306232 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:29:22.307204 29650 external_mini_cluster.cc:1428] Setting key 3b0e9a482660873c6f0215e90e17c42c
I20260929 20:29:22.308449 1880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:22.324905 1880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5528391355604b43a374cb7c078210d8: Bootstrap starting.
I20260929 20:29:22.325030 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:22.310092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38216 (local address 127.28.244.190:33071)
0929 20:29:22.317411 (+ 7319us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:22.317435 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:22.317516 (+ 81us) server_negotiation.cc:407] Connection header received
0929 20:29:22.320637 (+ 3121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:22.320679 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:22.321274 (+ 595us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:22.321786 (+ 512us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:22.323052 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:22.323060 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:22.323076 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:29:22.323091 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:22.323153 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:29:22.323169 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:22.323442 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:22.323594 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 20:29:22.323943 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6532,"thread_start_us":6216,"threads_started":1}
I20260929 20:29:22.329016 1880 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5528391355604b43a374cb7c078210d8: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:22.330170 1880 log.cc:824] T 00000000000000000000000000000000 P 5528391355604b43a374cb7c078210d8: Log is configured to *not* fsync() on all Append() calls
I20260929 20:29:22.333459 1880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5528391355604b43a374cb7c078210d8: No bootstrap required, opened a new log
I20260929 20:29:22.342123 1880 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5528391355604b43a374cb7c078210d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5528391355604b43a374cb7c078210d8" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 33071 } }
I20260929 20:29:22.342612 1880 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5528391355604b43a374cb7c078210d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:22.342713 1880 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5528391355604b43a374cb7c078210d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5528391355604b43a374cb7c078210d8, State: Initialized, Role: FOLLOWER
I20260929 20:29:22.343432 1880 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5528391355604b43a374cb7c078210d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5528391355604b43a374cb7c078210d8" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 33071 } }
I20260929 20:29:22.343700 1880 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5528391355604b43a374cb7c078210d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:29:22.343914 1880 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5528391355604b43a374cb7c078210d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:29:22.344120 1880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5528391355604b43a374cb7c078210d8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:22.347074 1880 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5528391355604b43a374cb7c078210d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5528391355604b43a374cb7c078210d8" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 33071 } }
I20260929 20:29:22.347725 1880 leader_election.cc:302] T 00000000000000000000000000000000 P 5528391355604b43a374cb7c078210d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5528391355604b43a374cb7c078210d8; no voters:
I20260929 20:29:22.348464 1880 leader_election.cc:288] T 00000000000000000000000000000000 P 5528391355604b43a374cb7c078210d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:29:22.348667 1885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5528391355604b43a374cb7c078210d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:29:22.349491 1885 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5528391355604b43a374cb7c078210d8 [term 1 LEADER]: Becoming Leader. State: Replica: 5528391355604b43a374cb7c078210d8, State: Running, Role: LEADER
I20260929 20:29:22.350104 1885 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5528391355604b43a374cb7c078210d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5528391355604b43a374cb7c078210d8" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 33071 } }
I20260929 20:29:22.350821 1880 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5528391355604b43a374cb7c078210d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:29:22.353461 1887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5528391355604b43a374cb7c078210d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5528391355604b43a374cb7c078210d8. Latest consensus state: current_term: 1 leader_uuid: "5528391355604b43a374cb7c078210d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5528391355604b43a374cb7c078210d8" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 33071 } } }
I20260929 20:29:22.354002 1887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5528391355604b43a374cb7c078210d8 [sys.catalog]: This master's current role is: LEADER
I20260929 20:29:22.354243 1886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5528391355604b43a374cb7c078210d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5528391355604b43a374cb7c078210d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5528391355604b43a374cb7c078210d8" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 33071 } } }
I20260929 20:29:22.354648 1886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5528391355604b43a374cb7c078210d8 [sys.catalog]: This master's current role is: LEADER
I20260929 20:29:22.357218 1890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:29:22.362834 1890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:29:22.372473 1890 catalog_manager.cc:1347] Generated new cluster ID: d797a800245740dcad69766b8b987384
I20260929 20:29:22.372617 1890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:29:22.398522 1890 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:29:22.399036 1890 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:29:22.406400 1890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5528391355604b43a374cb7c078210d8: Generated new TSK 0
I20260929 20:29:22.407321 1890 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:29:22.416534 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.129:0
--local_ip_for_outbound_sockets=127.28.244.129
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:33071
--builtin_ntp_servers=127.28.244.148:45555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 20:29:22.788610 1904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:22.788925 1904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:22.789076 1904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:22.796830 1904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:22.796967 1904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:22.797164 1904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.129
I20260929 20:29:22.806871 1904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:33071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.129
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:22.808415 1904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:22.810307 1904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:22.826246 1909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:22.826915 1912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:22.826956 1910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:22.827515 1904 server_base.cc:1034] running on GCE node
I20260929 20:29:22.828303 1904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:22.829523 1904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:22.830760 1904 hybrid_clock.cc:648] HybridClock initialized: now 1790713762830716 us; error 44 us; skew 500 ppm
I20260929 20:29:22.831177 1904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:29:22.834038 1904 webserver.cc:460] Webserver started at http://127.28.244.129:39781/ using document root <none> and password file <none>
I20260929 20:29:22.834827 1904 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:22.834957 1904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:22.835314 1904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:22.838622 1904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-0/data/instance:
uuid: "b97d4add634842349ad94c78d165880e"
format_stamp: "Formatted at 2026-09-29 20:29:22 on dist-test-slave-p29l"
server_key: "95e047aa961285de41869f89fb0eabba"
server_key_iv: "5f47b760c9bea26c80d74baf401d9b44"
server_key_version: "encryptionkey@0"
I20260929 20:29:22.839529 1904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance:
uuid: "b97d4add634842349ad94c78d165880e"
format_stamp: "Formatted at 2026-09-29 20:29:22 on dist-test-slave-p29l"
server_key: "95e047aa961285de41869f89fb0eabba"
server_key_iv: "5f47b760c9bea26c80d74baf401d9b44"
server_key_version: "encryptionkey@0"
I20260929 20:29:22.846451 1904 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 20:29:22.850732 1918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:22.852630 1904 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260929 20:29:22.852837 1904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-0/wal
uuid: "b97d4add634842349ad94c78d165880e"
format_stamp: "Formatted at 2026-09-29 20:29:22 on dist-test-slave-p29l"
server_key: "95e047aa961285de41869f89fb0eabba"
server_key_iv: "5f47b760c9bea26c80d74baf401d9b44"
server_key_version: "encryptionkey@0"
I20260929 20:29:22.853097 1904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:22.859378 1904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:22.860400 1904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:22.860692 1904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:22.861711 1904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:29:22.863283 1904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:29:22.863377 1904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:22.863536 1904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:29:22.863632 1904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:22.898788 1904 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.129:46339
I20260929 20:29:22.898844 2030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.129:46339 every 8 connection(s)
I20260929 20:29:22.900315 1904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
I20260929 20:29:22.909767 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 1904
I20260929 20:29:22.909938 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance
I20260929 20:29:22.910538 29650 external_mini_cluster.cc:1428] Setting key bfca6d80bc38aff46bacb5a3d1248190
I20260929 20:29:22.913625 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.130:0
--local_ip_for_outbound_sockets=127.28.244.130
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:33071
--builtin_ntp_servers=127.28.244.148:45555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 20:29:22.916553 2033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:22.903300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:53113 (local address 127.28.244.190:33071)
0929 20:29:22.903888 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:22.903896 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:22.905241 (+ 1345us) server_negotiation.cc:407] Connection header received
0929 20:29:22.907101 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:22.907106 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:22.907167 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:22.907272 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:22.915429 (+ 8157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:22.915440 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:22.915448 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:29:22.915478 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:22.915543 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:29:22.915565 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:22.915721 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:22.916099 (+ 378us) server_negotiation.cc:299] Negotiation successful
0929 20:29:22.916178 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":296,"threads_started":1}
I20260929 20:29:22.917714 2034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:22.903981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:33071 (local address 127.28.244.129:53113)
0929 20:29:22.905021 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:22.905072 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:22.906791 (+ 1719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:22.907509 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:22.907519 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:22.907627 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:22.914966 (+ 7339us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:29:22.914970 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:22.915027 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:29:22.915162 (+ 135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:22.915725 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:22.915739 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:22.915747 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 20:29:22.916150 (+ 403us) client_negotiation.cc:240] Negotiation successful
0929 20:29:22.916434 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":293,"threads_started":1}
I20260929 20:29:22.919629 2031 heartbeater.cc:347] Connected to a master server at 127.28.244.190:33071
I20260929 20:29:22.920042 2031 heartbeater.cc:464] Registering TS with master...
I20260929 20:29:22.920751 2031 heartbeater.cc:511] Master 127.28.244.190:33071 requested a full tablet report, sending...
I20260929 20:29:22.923241 1845 ts_manager.cc:194] Registered new tserver with Master: b97d4add634842349ad94c78d165880e (127.28.244.129:46339)
W20260929 20:29:23.284387 2035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:23.284765 2035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:23.284916 2035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:23.292817 2035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:23.292956 2035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:23.293072 2035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.130
I20260929 20:29:23.303339 2035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:33071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.130
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:23.304860 2035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:23.306926 2035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:23.322026 2041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:23.321987 2043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:23.322059 2040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:23.323025 2035 server_base.cc:1034] running on GCE node
I20260929 20:29:23.323794 2035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:23.324980 2035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:23.326208 2035 hybrid_clock.cc:648] HybridClock initialized: now 1790713763326150 us; error 53 us; skew 500 ppm
I20260929 20:29:23.326653 2035 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:29:23.329253 2035 webserver.cc:460] Webserver started at http://127.28.244.130:43173/ using document root <none> and password file <none>
I20260929 20:29:23.330236 2035 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:23.330338 2035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:23.330833 2035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:23.334107 2035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-1/data/instance:
uuid: "734255571c6049c580ab4c895d8af436"
format_stamp: "Formatted at 2026-09-29 20:29:23 on dist-test-slave-p29l"
server_key: "0294acc31c2412ed5f7cd7d1446b0eb6"
server_key_iv: "63cc69fd3baa2b558f2bb1dbc04cbcec"
server_key_version: "encryptionkey@0"
I20260929 20:29:23.335052 2035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance:
uuid: "734255571c6049c580ab4c895d8af436"
format_stamp: "Formatted at 2026-09-29 20:29:23 on dist-test-slave-p29l"
server_key: "0294acc31c2412ed5f7cd7d1446b0eb6"
server_key_iv: "63cc69fd3baa2b558f2bb1dbc04cbcec"
server_key_version: "encryptionkey@0"
I20260929 20:29:23.341166 2035 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 20:29:23.345423 2049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:23.347329 2035 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 20:29:23.347586 2035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-1/wal
uuid: "734255571c6049c580ab4c895d8af436"
format_stamp: "Formatted at 2026-09-29 20:29:23 on dist-test-slave-p29l"
server_key: "0294acc31c2412ed5f7cd7d1446b0eb6"
server_key_iv: "63cc69fd3baa2b558f2bb1dbc04cbcec"
server_key_version: "encryptionkey@0"
I20260929 20:29:23.347918 2035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:23.354981 2035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:23.356029 2035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:23.356354 2035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:23.357416 2035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:29:23.359117 2035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:29:23.359241 2035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:23.359383 2035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:29:23.359483 2035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:23.395327 2035 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.130:34835
I20260929 20:29:23.395473 2161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.130:34835 every 8 connection(s)
I20260929 20:29:23.396801 2035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
I20260929 20:29:23.397363 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 2035
I20260929 20:29:23.397555 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance
I20260929 20:29:23.398222 29650 external_mini_cluster.cc:1428] Setting key 28be86e9360e38c77556fdfb6e41249c
I20260929 20:29:23.401753 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.131:0
--local_ip_for_outbound_sockets=127.28.244.131
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:33071
--builtin_ntp_servers=127.28.244.148:45555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 20:29:23.406224 2033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:23.399789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:59799 (local address 127.28.244.190:33071)
0929 20:29:23.400003 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:23.400011 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:23.401758 (+ 1747us) server_negotiation.cc:407] Connection header received
0929 20:29:23.403622 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:23.403628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:23.403720 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:23.403853 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:23.405433 (+ 1580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:23.405439 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:23.405442 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:29:23.405462 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:23.405505 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:29:23.405517 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:23.405666 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:23.405857 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 20:29:23.405936 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 20:29:23.407511 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:23.400523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:33071 (local address 127.28.244.130:59799)
0929 20:29:23.401590 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:23.401634 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:23.403309 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:23.404147 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:23.404158 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:23.404270 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:23.405076 (+ 806us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:29:23.405078 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:23.405125 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:29:23.405226 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:23.405615 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:23.405629 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:23.405640 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 20:29:23.405951 (+ 311us) client_negotiation.cc:240] Negotiation successful
0929 20:29:23.406276 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":342,"threads_started":1}
I20260929 20:29:23.409145 2162 heartbeater.cc:347] Connected to a master server at 127.28.244.190:33071
I20260929 20:29:23.409498 2162 heartbeater.cc:464] Registering TS with master...
I20260929 20:29:23.410169 2162 heartbeater.cc:511] Master 127.28.244.190:33071 requested a full tablet report, sending...
I20260929 20:29:23.411705 1845 ts_manager.cc:194] Registered new tserver with Master: 734255571c6049c580ab4c895d8af436 (127.28.244.130:34835)
W20260929 20:29:23.769534 2165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:23.769937 2165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:23.770117 2165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:23.779031 2165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:23.779201 2165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:23.779400 2165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.131
I20260929 20:29:23.790681 2165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:33071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.131
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:23.792326 2165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:23.794384 2165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:23.810258 2170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:23.810204 2171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:23.810137 2173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:23.811358 2165 server_base.cc:1034] running on GCE node
I20260929 20:29:23.812155 2165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:23.813338 2165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:23.814648 2165 hybrid_clock.cc:648] HybridClock initialized: now 1790713763814587 us; error 53 us; skew 500 ppm
I20260929 20:29:23.815054 2165 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:29:23.817930 2165 webserver.cc:460] Webserver started at http://127.28.244.131:44671/ using document root <none> and password file <none>
I20260929 20:29:23.818764 2165 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:23.818930 2165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:23.819308 2165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:23.822655 2165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-2/data/instance:
uuid: "fe1cdd57d8e8474b8f91f82ee004757b"
format_stamp: "Formatted at 2026-09-29 20:29:23 on dist-test-slave-p29l"
server_key: "c9ccd19d97c8ce273316de71fd91b0e5"
server_key_iv: "bbbd7584a4059374e07d8f4dc360b27e"
server_key_version: "encryptionkey@0"
I20260929 20:29:23.823632 2165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance:
uuid: "fe1cdd57d8e8474b8f91f82ee004757b"
format_stamp: "Formatted at 2026-09-29 20:29:23 on dist-test-slave-p29l"
server_key: "c9ccd19d97c8ce273316de71fd91b0e5"
server_key_iv: "bbbd7584a4059374e07d8f4dc360b27e"
server_key_version: "encryptionkey@0"
I20260929 20:29:23.829768 2165 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 20:29:23.834388 2179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:23.836016 2165 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:29:23.836231 2165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-2/wal
uuid: "fe1cdd57d8e8474b8f91f82ee004757b"
format_stamp: "Formatted at 2026-09-29 20:29:23 on dist-test-slave-p29l"
server_key: "c9ccd19d97c8ce273316de71fd91b0e5"
server_key_iv: "bbbd7584a4059374e07d8f4dc360b27e"
server_key_version: "encryptionkey@0"
I20260929 20:29:23.836505 2165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:23.843017 2165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:23.844051 2165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:23.844357 2165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:23.845402 2165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:29:23.847023 2165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:29:23.847150 2165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:23.847298 2165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:29:23.847394 2165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:23.885423 2165 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.131:44483
I20260929 20:29:23.885527 2291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.131:44483 every 8 connection(s)
I20260929 20:29:23.886909 2165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
I20260929 20:29:23.894905 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 2165
I20260929 20:29:23.895117 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance
I20260929 20:29:23.895676 29650 external_mini_cluster.cc:1428] Setting key e3e6fbb7bde2e40d193cf45bd7bb9acf
I20260929 20:29:23.896580 2033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:23.889986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:33785 (local address 127.28.244.190:33071)
0929 20:29:23.890241 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:23.890246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:23.891951 (+ 1705us) server_negotiation.cc:407] Connection header received
0929 20:29:23.893809 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:23.893814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:23.893876 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:23.893989 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:23.895732 (+ 1743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:23.895739 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:23.895743 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:29:23.895769 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:23.895841 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:29:23.895862 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:23.896011 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:23.896221 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 20:29:23.896305 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260929 20:29:23.897920 2294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:23.890595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:33071 (local address 127.28.244.131:33785)
0929 20:29:23.891729 (+ 1134us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:23.891770 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:23.893484 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:23.894277 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:23.894286 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:23.894451 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:23.895338 (+ 887us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:29:23.895342 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:23.895395 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:29:23.895502 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:23.895979 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:23.895988 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:23.895995 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 20:29:23.896264 (+ 269us) client_negotiation.cc:240] Negotiation successful
0929 20:29:23.896619 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":335,"threads_started":1}
I20260929 20:29:23.900341 2292 heartbeater.cc:347] Connected to a master server at 127.28.244.190:33071
I20260929 20:29:23.900776 2292 heartbeater.cc:464] Registering TS with master...
I20260929 20:29:23.901530 2292 heartbeater.cc:511] Master 127.28.244.190:33071 requested a full tablet report, sending...
I20260929 20:29:23.903522 1845 ts_manager.cc:194] Registered new tserver with Master: fe1cdd57d8e8474b8f91f82ee004757b (127.28.244.131:44483)
I20260929 20:29:23.913280 29650 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:29:23.921559 2033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:23.918429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38228 (local address 127.28.244.190:33071)
0929 20:29:23.918730 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:23.918740 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:23.919178 (+ 438us) server_negotiation.cc:407] Connection header received
0929 20:29:23.919427 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:23.919432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:23.919504 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:23.919626 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:23.920878 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:23.920883 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:23.920887 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:29:23.920906 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:23.920944 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:29:23.920957 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:23.921110 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:23.921193 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 20:29:23.921266 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260929 20:29:23.926782 2031 heartbeater.cc:503] Master 127.28.244.190:33071 was elected leader, sending a full tablet report...
I20260929 20:29:23.926086 1845 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38228:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:29:23.933969 1845 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:29:23.954445 2304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:23.947987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:44483 (local address 127.0.0.1:39912)
0929 20:29:23.950665 (+ 2678us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:23.950684 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:23.950788 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:23.952114 (+ 1326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:23.952121 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:23.952152 (+ 31us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:23.952946 (+ 794us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:29:23.952948 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:23.952969 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:29:23.953063 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:23.953625 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:23.953641 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:23.953649 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 20:29:23.953835 (+ 186us) client_negotiation.cc:240] Negotiation successful
0929 20:29:23.953959 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2544,"thread_start_us":1008,"threads_started":1}
I20260929 20:29:23.954424 2306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:23.948098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56370 (local address 127.28.244.129:46339)
0929 20:29:23.948781 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:23.948793 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:23.949723 (+ 930us) server_negotiation.cc:407] Connection header received
0929 20:29:23.950084 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:23.950096 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:23.950359 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:23.950519 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:23.953264 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:23.953274 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:23.953291 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:29:23.953311 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:23.953387 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:29:23.953408 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:23.953563 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:23.953918 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 20:29:23.954038 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":777,"threads_started":1}
I20260929 20:29:23.955281 2305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:23.947792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:46339 (local address 127.0.0.1:56370)
0929 20:29:23.949511 (+ 1719us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:23.949566 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:23.949891 (+ 325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:23.950739 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:23.950747 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:23.950850 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:23.952917 (+ 2067us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:29:23.952923 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:23.952966 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:29:23.953063 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:23.953648 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:23.953652 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:23.953655 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 20:29:23.954792 (+ 1137us) client_negotiation.cc:240] Negotiation successful
0929 20:29:23.954853 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1314,"thread_start_us":2529,"threads_started":1}
I20260929 20:29:23.955448 2307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:23.948599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39912 (local address 127.28.244.131:44483)
0929 20:29:23.950434 (+ 1835us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:23.950445 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:23.950906 (+ 461us) server_negotiation.cc:407] Connection header received
0929 20:29:23.950998 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:23.951007 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:23.951306 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:23.951544 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:23.953265 (+ 1721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:23.953273 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:23.953291 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:29:23.953311 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:23.953374 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:29:23.953398 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:23.953567 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:23.954941 (+ 1374us) server_negotiation.cc:299] Negotiation successful
0929 20:29:23.955048 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1628,"thread_start_us":1423,"threads_started":1}
I20260929 20:29:23.960003 2305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:23.951433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:34835 (local address 127.0.0.1:47226)
0929 20:29:23.955568 (+ 4135us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:23.955591 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:23.955700 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:23.957273 (+ 1573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:23.957279 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:23.957317 (+ 38us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:23.957968 (+ 651us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:29:23.957971 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:23.957991 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:29:23.958046 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:23.959446 (+ 1400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:23.959451 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:23.959458 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 20:29:23.959620 (+ 162us) client_negotiation.cc:240] Negotiation successful
0929 20:29:23.959685 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4014,"thread_start_us":4618,"threads_started":1}
I20260929 20:29:23.960284 2227 tablet_service.cc:1467] Processing CreateTablet for tablet c25238c69fbb47f9829cae5e7dc92387 (DEFAULT_TABLE table=test-table [id=57381461dcd84939a8d31e6308c3a3cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:29:23.961064 2308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:23.951115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47226 (local address 127.28.244.130:34835)
0929 20:29:23.956613 (+ 5498us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:23.956623 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:23.956657 (+ 34us) server_negotiation.cc:407] Connection header received
0929 20:29:23.956766 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:23.956775 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:23.957039 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:23.957689 (+ 650us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:23.959166 (+ 1477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:23.959178 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:23.959193 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:29:23.959218 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:23.959278 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:29:23.959305 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:23.959473 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:23.960627 (+ 1154us) server_negotiation.cc:299] Negotiation successful
0929 20:29:23.960741 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5315,"thread_start_us":5176,"threads_started":1}
I20260929 20:29:23.961928 2227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c25238c69fbb47f9829cae5e7dc92387. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:23.961326 1966 tablet_service.cc:1467] Processing CreateTablet for tablet c25238c69fbb47f9829cae5e7dc92387 (DEFAULT_TABLE table=test-table [id=57381461dcd84939a8d31e6308c3a3cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:29:23.962713 1966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c25238c69fbb47f9829cae5e7dc92387. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:23.965312 2097 tablet_service.cc:1467] Processing CreateTablet for tablet c25238c69fbb47f9829cae5e7dc92387 (DEFAULT_TABLE table=test-table [id=57381461dcd84939a8d31e6308c3a3cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:29:23.967006 2097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c25238c69fbb47f9829cae5e7dc92387. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:23.975849 2310 tablet_bootstrap.cc:492] T c25238c69fbb47f9829cae5e7dc92387 P b97d4add634842349ad94c78d165880e: Bootstrap starting.
I20260929 20:29:23.980630 2310 tablet_bootstrap.cc:654] T c25238c69fbb47f9829cae5e7dc92387 P b97d4add634842349ad94c78d165880e: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:23.980680 2311 tablet_bootstrap.cc:492] T c25238c69fbb47f9829cae5e7dc92387 P fe1cdd57d8e8474b8f91f82ee004757b: Bootstrap starting.
I20260929 20:29:23.982395 2310 log.cc:824] T c25238c69fbb47f9829cae5e7dc92387 P b97d4add634842349ad94c78d165880e: Log is configured to *not* fsync() on all Append() calls
I20260929 20:29:23.983461 2312 tablet_bootstrap.cc:492] T c25238c69fbb47f9829cae5e7dc92387 P 734255571c6049c580ab4c895d8af436: Bootstrap starting.
I20260929 20:29:23.985998 2311 tablet_bootstrap.cc:654] T c25238c69fbb47f9829cae5e7dc92387 P fe1cdd57d8e8474b8f91f82ee004757b: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:23.987284 2312 tablet_bootstrap.cc:654] T c25238c69fbb47f9829cae5e7dc92387 P 734255571c6049c580ab4c895d8af436: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:23.987353 2311 log.cc:824] T c25238c69fbb47f9829cae5e7dc92387 P fe1cdd57d8e8474b8f91f82ee004757b: Log is configured to *not* fsync() on all Append() calls
I20260929 20:29:23.988667 2312 log.cc:824] T c25238c69fbb47f9829cae5e7dc92387 P 734255571c6049c580ab4c895d8af436: Log is configured to *not* fsync() on all Append() calls
I20260929 20:29:23.988793 2310 tablet_bootstrap.cc:492] T c25238c69fbb47f9829cae5e7dc92387 P b97d4add634842349ad94c78d165880e: No bootstrap required, opened a new log
I20260929 20:29:23.989174 2310 ts_tablet_manager.cc:1397] T c25238c69fbb47f9829cae5e7dc92387 P b97d4add634842349ad94c78d165880e: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.005s
I20260929 20:29:23.991086 2311 tablet_bootstrap.cc:492] T c25238c69fbb47f9829cae5e7dc92387 P fe1cdd57d8e8474b8f91f82ee004757b: No bootstrap required, opened a new log
I20260929 20:29:23.991379 2311 ts_tablet_manager.cc:1397] T c25238c69fbb47f9829cae5e7dc92387 P fe1cdd57d8e8474b8f91f82ee004757b: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260929 20:29:23.991389 2312 tablet_bootstrap.cc:492] T c25238c69fbb47f9829cae5e7dc92387 P 734255571c6049c580ab4c895d8af436: No bootstrap required, opened a new log
I20260929 20:29:23.991679 2312 ts_tablet_manager.cc:1397] T c25238c69fbb47f9829cae5e7dc92387 P 734255571c6049c580ab4c895d8af436: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260929 20:29:23.999486 2312 raft_consensus.cc:348] T c25238c69fbb47f9829cae5e7dc92387 P 734255571c6049c580ab4c895d8af436 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97d4add634842349ad94c78d165880e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 46339 } } peers { permanent_uuid: "fe1cdd57d8e8474b8f91f82ee004757b" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44483 } } peers { permanent_uuid: "734255571c6049c580ab4c895d8af436" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34835 } }
I20260929 20:29:24.000032 2312 raft_consensus.cc:374] T c25238c69fbb47f9829cae5e7dc92387 P 734255571c6049c580ab4c895d8af436 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:24.000139 2312 raft_consensus.cc:729] T c25238c69fbb47f9829cae5e7dc92387 P 734255571c6049c580ab4c895d8af436 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 734255571c6049c580ab4c895d8af436, State: Initialized, Role: FOLLOWER
I20260929 20:29:24.000916 2312 consensus_queue.cc:260] T c25238c69fbb47f9829cae5e7dc92387 P 734255571c6049c580ab4c895d8af436 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97d4add634842349ad94c78d165880e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 46339 } } peers { permanent_uuid: "fe1cdd57d8e8474b8f91f82ee004757b" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44483 } } peers { permanent_uuid: "734255571c6049c580ab4c895d8af436" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34835 } }
I20260929 20:29:24.001101 2310 raft_consensus.cc:348] T c25238c69fbb47f9829cae5e7dc92387 P b97d4add634842349ad94c78d165880e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97d4add634842349ad94c78d165880e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 46339 } } peers { permanent_uuid: "fe1cdd57d8e8474b8f91f82ee004757b" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44483 } } peers { permanent_uuid: "734255571c6049c580ab4c895d8af436" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34835 } }
I20260929 20:29:24.001598 2310 raft_consensus.cc:374] T c25238c69fbb47f9829cae5e7dc92387 P b97d4add634842349ad94c78d165880e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:24.001694 2310 raft_consensus.cc:729] T c25238c69fbb47f9829cae5e7dc92387 P b97d4add634842349ad94c78d165880e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b97d4add634842349ad94c78d165880e, State: Initialized, Role: FOLLOWER
I20260929 20:29:24.002508 2310 consensus_queue.cc:260] T c25238c69fbb47f9829cae5e7dc92387 P b97d4add634842349ad94c78d165880e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97d4add634842349ad94c78d165880e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 46339 } } peers { permanent_uuid: "fe1cdd57d8e8474b8f91f82ee004757b" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44483 } } peers { permanent_uuid: "734255571c6049c580ab4c895d8af436" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34835 } }
I20260929 20:29:24.002651 2311 raft_consensus.cc:348] T c25238c69fbb47f9829cae5e7dc92387 P fe1cdd57d8e8474b8f91f82ee004757b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97d4add634842349ad94c78d165880e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 46339 } } peers { permanent_uuid: "fe1cdd57d8e8474b8f91f82ee004757b" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44483 } } peers { permanent_uuid: "734255571c6049c580ab4c895d8af436" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34835 } }
I20260929 20:29:24.003901 2311 raft_consensus.cc:374] T c25238c69fbb47f9829cae5e7dc92387 P fe1cdd57d8e8474b8f91f82ee004757b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:24.004053 2311 raft_consensus.cc:729] T c25238c69fbb47f9829cae5e7dc92387 P fe1cdd57d8e8474b8f91f82ee004757b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe1cdd57d8e8474b8f91f82ee004757b, State: Initialized, Role: FOLLOWER
I20260929 20:29:24.004118 2162 heartbeater.cc:503] Master 127.28.244.190:33071 was elected leader, sending a full tablet report...
I20260929 20:29:24.005030 2311 consensus_queue.cc:260] T c25238c69fbb47f9829cae5e7dc92387 P fe1cdd57d8e8474b8f91f82ee004757b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97d4add634842349ad94c78d165880e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 46339 } } peers { permanent_uuid: "fe1cdd57d8e8474b8f91f82ee004757b" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44483 } } peers { permanent_uuid: "734255571c6049c580ab4c895d8af436" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34835 } }
I20260929 20:29:24.005400 2312 ts_tablet_manager.cc:1428] T c25238c69fbb47f9829cae5e7dc92387 P 734255571c6049c580ab4c895d8af436: Time spent starting tablet: real 0.014s user 0.012s sys 0.002s
I20260929 20:29:24.006577 2292 heartbeater.cc:503] Master 127.28.244.190:33071 was elected leader, sending a full tablet report...
I20260929 20:29:24.007529 2311 ts_tablet_manager.cc:1428] T c25238c69fbb47f9829cae5e7dc92387 P fe1cdd57d8e8474b8f91f82ee004757b: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260929 20:29:24.009370 2310 ts_tablet_manager.cc:1428] T c25238c69fbb47f9829cae5e7dc92387 P b97d4add634842349ad94c78d165880e: Time spent starting tablet: real 0.020s user 0.014s sys 0.006s
W20260929 20:29:24.138964 2293 tablet.cc:2369] T c25238c69fbb47f9829cae5e7dc92387 P fe1cdd57d8e8474b8f91f82ee004757b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:29:24.152465 2163 tablet.cc:2369] T c25238c69fbb47f9829cae5e7dc92387 P 734255571c6049c580ab4c895d8af436: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:29:24.152936 2032 tablet.cc:2369] T c25238c69fbb47f9829cae5e7dc92387 P b97d4add634842349ad94c78d165880e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:29:24.177835 2318 raft_consensus.cc:482] T c25238c69fbb47f9829cae5e7dc92387 P fe1cdd57d8e8474b8f91f82ee004757b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:29:24.178248 2318 raft_consensus.cc:504] T c25238c69fbb47f9829cae5e7dc92387 P fe1cdd57d8e8474b8f91f82ee004757b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97d4add634842349ad94c78d165880e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 46339 } } peers { permanent_uuid: "fe1cdd57d8e8474b8f91f82ee004757b" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44483 } } peers { permanent_uuid: "734255571c6049c580ab4c895d8af436" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34835 } }
I20260929 20:29:24.180287 2318 leader_election.cc:288] T c25238c69fbb47f9829cae5e7dc92387 P fe1cdd57d8e8474b8f91f82ee004757b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b97d4add634842349ad94c78d165880e (127.28.244.129:46339), 734255571c6049c580ab4c895d8af436 (127.28.244.130:34835)
I20260929 20:29:24.183168 2306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:24.180736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:34547 (local address 127.28.244.129:46339)
0929 20:29:24.180983 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:24.180988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:24.181093 (+ 105us) server_negotiation.cc:407] Connection header received
0929 20:29:24.181296 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:24.181306 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:24.181382 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:24.181525 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:24.182545 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:24.182551 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:24.182555 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:29:24.182576 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:24.182618 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:29:24.182635 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:24.182767 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:24.182887 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 20:29:24.182945 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260929 20:29:24.183336 2294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:24.180730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:46339 (local address 127.28.244.131:34547)
0929 20:29:24.180975 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:24.181002 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:24.181140 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:24.181564 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:24.181572 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:24.181601 (+ 29us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:24.182291 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:29:24.182298 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:24.182322 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:29:24.182374 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:24.182744 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:24.182749 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:24.182754 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 20:29:24.182876 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 20:29:24.182997 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260929 20:29:24.184609 2308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:24.181457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:37037 (local address 127.28.244.130:34835)
0929 20:29:24.181676 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:24.181686 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:24.182397 (+ 711us) server_negotiation.cc:407] Connection header received
0929 20:29:24.182542 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:24.182547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:24.182607 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:24.182718 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:24.183967 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:24.183973 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:24.183978 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:29:24.184003 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:24.184035 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:29:24.184047 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:24.184142 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:24.184345 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 20:29:24.184406 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 20:29:24.184554 1986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c25238c69fbb47f9829cae5e7dc92387" candidate_uuid: "fe1cdd57d8e8474b8f91f82ee004757b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b97d4add634842349ad94c78d165880e" is_pre_election: true
I20260929 20:29:24.184903 2319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:24.181265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:34835 (local address 127.28.244.131:37037)
0929 20:29:24.182249 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:24.182277 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:24.182396 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:24.182758 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:24.182762 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:24.182787 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:24.183718 (+ 931us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:29:24.183721 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:24.183735 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:29:24.183775 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:24.184158 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:24.184162 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:24.184167 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 20:29:24.184314 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 20:29:24.184377 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":806,"thread_start_us":538,"threads_started":1}
I20260929 20:29:24.185240 1986 raft_consensus.cc:2454] T c25238c69fbb47f9829cae5e7dc92387 P b97d4add634842349ad94c78d165880e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe1cdd57d8e8474b8f91f82ee004757b in term 0.
I20260929 20:29:24.185976 2181 leader_election.cc:302] T c25238c69fbb47f9829cae5e7dc92387 P fe1cdd57d8e8474b8f91f82ee004757b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b97d4add634842349ad94c78d165880e, fe1cdd57d8e8474b8f91f82ee004757b; no voters:
I20260929 20:29:24.185967 2117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c25238c69fbb47f9829cae5e7dc92387" candidate_uuid: "fe1cdd57d8e8474b8f91f82ee004757b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "734255571c6049c580ab4c895d8af436" is_pre_election: true
I20260929 20:29:24.186409 2318 raft_consensus.cc:2781] T c25238c69fbb47f9829cae5e7dc92387 P fe1cdd57d8e8474b8f91f82ee004757b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:29:24.186585 2318 raft_consensus.cc:482] T c25238c69fbb47f9829cae5e7dc92387 P fe1cdd57d8e8474b8f91f82ee004757b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:29:24.186601 2117 raft_consensus.cc:2454] T c25238c69fbb47f9829cae5e7dc92387 P 734255571c6049c580ab4c895d8af436 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe1cdd57d8e8474b8f91f82ee004757b in term 0.
I20260929 20:29:24.186714 2318 raft_consensus.cc:3037] T c25238c69fbb47f9829cae5e7dc92387 P fe1cdd57d8e8474b8f91f82ee004757b [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:24.189360 2318 raft_consensus.cc:504] T c25238c69fbb47f9829cae5e7dc92387 P fe1cdd57d8e8474b8f91f82ee004757b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97d4add634842349ad94c78d165880e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 46339 } } peers { permanent_uuid: "fe1cdd57d8e8474b8f91f82ee004757b" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44483 } } peers { permanent_uuid: "734255571c6049c580ab4c895d8af436" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34835 } }
I20260929 20:29:24.189971 2318 leader_election.cc:288] T c25238c69fbb47f9829cae5e7dc92387 P fe1cdd57d8e8474b8f91f82ee004757b [CANDIDATE]: Term 1 election: Requested vote from peers b97d4add634842349ad94c78d165880e (127.28.244.129:46339), 734255571c6049c580ab4c895d8af436 (127.28.244.130:34835)
I20260929 20:29:24.190584 1986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c25238c69fbb47f9829cae5e7dc92387" candidate_uuid: "fe1cdd57d8e8474b8f91f82ee004757b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b97d4add634842349ad94c78d165880e"
I20260929 20:29:24.190611 2117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c25238c69fbb47f9829cae5e7dc92387" candidate_uuid: "fe1cdd57d8e8474b8f91f82ee004757b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "734255571c6049c580ab4c895d8af436"
I20260929 20:29:24.190840 2117 raft_consensus.cc:3037] T c25238c69fbb47f9829cae5e7dc92387 P 734255571c6049c580ab4c895d8af436 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:24.190811 1986 raft_consensus.cc:3037] T c25238c69fbb47f9829cae5e7dc92387 P b97d4add634842349ad94c78d165880e [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:24.193318 2117 raft_consensus.cc:2454] T c25238c69fbb47f9829cae5e7dc92387 P 734255571c6049c580ab4c895d8af436 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe1cdd57d8e8474b8f91f82ee004757b in term 1.
I20260929 20:29:24.193427 1986 raft_consensus.cc:2454] T c25238c69fbb47f9829cae5e7dc92387 P b97d4add634842349ad94c78d165880e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe1cdd57d8e8474b8f91f82ee004757b in term 1.
I20260929 20:29:24.193954 2181 leader_election.cc:302] T c25238c69fbb47f9829cae5e7dc92387 P fe1cdd57d8e8474b8f91f82ee004757b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 734255571c6049c580ab4c895d8af436, fe1cdd57d8e8474b8f91f82ee004757b; no voters:
I20260929 20:29:24.194324 2318 raft_consensus.cc:2781] T c25238c69fbb47f9829cae5e7dc92387 P fe1cdd57d8e8474b8f91f82ee004757b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:29:24.194841 2318 raft_consensus.cc:686] T c25238c69fbb47f9829cae5e7dc92387 P fe1cdd57d8e8474b8f91f82ee004757b [term 1 LEADER]: Becoming Leader. State: Replica: fe1cdd57d8e8474b8f91f82ee004757b, State: Running, Role: LEADER
I20260929 20:29:24.195495 2318 consensus_queue.cc:237] T c25238c69fbb47f9829cae5e7dc92387 P fe1cdd57d8e8474b8f91f82ee004757b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97d4add634842349ad94c78d165880e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 46339 } } peers { permanent_uuid: "fe1cdd57d8e8474b8f91f82ee004757b" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44483 } } peers { permanent_uuid: "734255571c6049c580ab4c895d8af436" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34835 } }
I20260929 20:29:24.200764 1844 catalog_manager.cc:5495] T c25238c69fbb47f9829cae5e7dc92387 P fe1cdd57d8e8474b8f91f82ee004757b reported cstate change: term changed from 0 to 1, leader changed from <none> to fe1cdd57d8e8474b8f91f82ee004757b (127.28.244.131). New cstate: current_term: 1 leader_uuid: "fe1cdd57d8e8474b8f91f82ee004757b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97d4add634842349ad94c78d165880e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 46339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe1cdd57d8e8474b8f91f82ee004757b" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "734255571c6049c580ab4c895d8af436" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34835 } health_report { overall_health: UNKNOWN } } }
I20260929 20:29:24.269485 2307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:24.267175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39916 (local address 127.28.244.131:44483)
0929 20:29:24.267397 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:24.267401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:24.267426 (+ 25us) server_negotiation.cc:407] Connection header received
0929 20:29:24.267699 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:24.267703 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:24.267793 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:24.267915 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:24.268836 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:24.268840 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:24.268843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:29:24.268859 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:24.268901 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:29:24.268912 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:24.269015 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:24.269118 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 20:29:24.269168 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 20:29:24.278855 1986 raft_consensus.cc:1261] T c25238c69fbb47f9829cae5e7dc92387 P b97d4add634842349ad94c78d165880e [term 1 FOLLOWER]: Refusing update from remote peer fe1cdd57d8e8474b8f91f82ee004757b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:29:24.278855 2117 raft_consensus.cc:1261] T c25238c69fbb47f9829cae5e7dc92387 P 734255571c6049c580ab4c895d8af436 [term 1 FOLLOWER]: Refusing update from remote peer fe1cdd57d8e8474b8f91f82ee004757b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:29:24.279938 2318 consensus_queue.cc:1035] T c25238c69fbb47f9829cae5e7dc92387 P fe1cdd57d8e8474b8f91f82ee004757b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b97d4add634842349ad94c78d165880e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 46339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:29:24.280316 2320 consensus_queue.cc:1035] T c25238c69fbb47f9829cae5e7dc92387 P fe1cdd57d8e8474b8f91f82ee004757b [LEADER]: Connected to new peer: Peer: permanent_uuid: "734255571c6049c580ab4c895d8af436" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:29:24.293668 2323 mvcc.cc:205] Tried to move back new op lower bound from 7334763578474762240 to 7334763578149556224. Current Snapshot: MvccSnapshot[applied={T|T < 7334763578474762240}]
I20260929 20:29:24.312023 1844 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:29:24.312407 1844 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:29:24.316674 1844 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5528391355604b43a374cb7c078210d8: Sending DeleteTablet for 3 replicas of tablet c25238c69fbb47f9829cae5e7dc92387
I20260929 20:29:24.318400 1966 tablet_service.cc:1514] Processing DeleteTablet for tablet c25238c69fbb47f9829cae5e7dc92387 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:29:24 UTC) from {username='slave'} at 127.0.0.1:56370
I20260929 20:29:24.318517 2097 tablet_service.cc:1514] Processing DeleteTablet for tablet c25238c69fbb47f9829cae5e7dc92387 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:29:24 UTC) from {username='slave'} at 127.0.0.1:47226
I20260929 20:29:24.319511 2227 tablet_service.cc:1514] Processing DeleteTablet for tablet c25238c69fbb47f9829cae5e7dc92387 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:29:24 UTC) from {username='slave'} at 127.0.0.1:39912
I20260929 20:29:24.320765 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 1904
I20260929 20:29:24.329162 2335 tablet_replica.cc:330] T c25238c69fbb47f9829cae5e7dc92387 P fe1cdd57d8e8474b8f91f82ee004757b: stopping tablet replica
I20260929 20:29:24.329957 2335 raft_consensus.cc:2229] T c25238c69fbb47f9829cae5e7dc92387 P fe1cdd57d8e8474b8f91f82ee004757b [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:29:24.330807 2335 raft_consensus.cc:2258] T c25238c69fbb47f9829cae5e7dc92387 P fe1cdd57d8e8474b8f91f82ee004757b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:29:24.332145 2306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:24.322739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56384 (local address 127.28.244.129:46339)
0929 20:29:24.322998 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:24.323007 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:24.328418 (+ 5411us) server_negotiation.cc:407] Connection header received
0929 20:29:24.328638 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:24.328647 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:24.328809 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:24.328953 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:24.330868 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:24.330879 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:24.330887 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:29:24.330923 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:24.330982 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:29:24.331005 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:24.331163 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:24.331605 (+ 442us) server_negotiation.cc:299] Negotiation successful
0929 20:29:24.331704 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 20:29:24.331825 2333 tablet_replica.cc:330] T c25238c69fbb47f9829cae5e7dc92387 P b97d4add634842349ad94c78d165880e: stopping tablet replica
I20260929 20:29:24.333081 2026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:29:24.335638 2335 ts_tablet_manager.cc:1905] T c25238c69fbb47f9829cae5e7dc92387 P fe1cdd57d8e8474b8f91f82ee004757b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:29:24.338780 2334 tablet_replica.cc:330] T c25238c69fbb47f9829cae5e7dc92387 P 734255571c6049c580ab4c895d8af436: stopping tablet replica
I20260929 20:29:24.339321 2334 raft_consensus.cc:2229] T c25238c69fbb47f9829cae5e7dc92387 P 734255571c6049c580ab4c895d8af436 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:29:24.339979 2334 raft_consensus.cc:2258] T c25238c69fbb47f9829cae5e7dc92387 P 734255571c6049c580ab4c895d8af436 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:29:24.342638 2334 ts_tablet_manager.cc:1905] T c25238c69fbb47f9829cae5e7dc92387 P 734255571c6049c580ab4c895d8af436: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:29:24.350670 2334 ts_tablet_manager.cc:1918] T c25238c69fbb47f9829cae5e7dc92387 P 734255571c6049c580ab4c895d8af436: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:29:24.350932 2334 log.cc:1196] T c25238c69fbb47f9829cae5e7dc92387 P 734255571c6049c580ab4c895d8af436: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-1/wal/wals/c25238c69fbb47f9829cae5e7dc92387
I20260929 20:29:24.351644 2334 ts_tablet_manager.cc:1939] T c25238c69fbb47f9829cae5e7dc92387 P 734255571c6049c580ab4c895d8af436: Deleting consensus metadata
I20260929 20:29:24.353086 2335 ts_tablet_manager.cc:1918] T c25238c69fbb47f9829cae5e7dc92387 P fe1cdd57d8e8474b8f91f82ee004757b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:29:24.353446 2335 log.cc:1196] T c25238c69fbb47f9829cae5e7dc92387 P fe1cdd57d8e8474b8f91f82ee004757b: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713721201454-29650-0/minicluster-data/ts-2/wal/wals/c25238c69fbb47f9829cae5e7dc92387
I20260929 20:29:24.354256 2335 ts_tablet_manager.cc:1939] T c25238c69fbb47f9829cae5e7dc92387 P fe1cdd57d8e8474b8f91f82ee004757b: Deleting consensus metadata
I20260929 20:29:24.354374 1830 catalog_manager.cc:4841] TS 734255571c6049c580ab4c895d8af436 (127.28.244.130:34835): tablet c25238c69fbb47f9829cae5e7dc92387 (table test-table [id=57381461dcd84939a8d31e6308c3a3cc]) successfully deleted
I20260929 20:29:24.356750 1829 catalog_manager.cc:4841] TS fe1cdd57d8e8474b8f91f82ee004757b (127.28.244.131:44483): tablet c25238c69fbb47f9829cae5e7dc92387 (table test-table [id=57381461dcd84939a8d31e6308c3a3cc]) successfully deleted
I20260929 20:29:24.333220 2333 raft_consensus.cc:2229] T c25238c69fbb47f9829cae5e7dc92387 P b97d4add634842349ad94c78d165880e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:29:24.458109 2333 raft_consensus.cc:2258] T c25238c69fbb47f9829cae5e7dc92387 P b97d4add634842349ad94c78d165880e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:29:24.458325 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 1904
I20260929 20:29:24.482630 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 2035
W20260929 20:29:24.482630 1830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.244.129:46339 (error 108)
I20260929 20:29:24.484949 2339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:24.484191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:46339 (local address 127.0.0.1:56392)
0929 20:29:24.484665 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:24.484784 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:46339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":259,"threads_started":1}
W20260929 20:29:24.485210 1830 catalog_manager.cc:4584] TS b97d4add634842349ad94c78d165880e (127.28.244.129:46339): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c25238c69fbb47f9829cae5e7dc92387: Network error: Client connection negotiation failed: client connection to 127.28.244.129:46339: connect: Connection refused (error 111)
I20260929 20:29:24.486539 2308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:24.483902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47234 (local address 127.28.244.130:34835)
0929 20:29:24.484107 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:24.484113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:24.484135 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:29:24.484206 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:24.484212 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:24.484324 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:24.484432 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:24.485734 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:24.485742 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:24.485746 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:29:24.485767 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:24.485809 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:29:24.485825 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:24.485967 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:24.486174 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 20:29:24.486245 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:29:24.487344 2157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:29:24.538926 2339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:24.538386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:46339 (local address 127.0.0.1:56394)
0929 20:29:24.538608 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:24.538748 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:46339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 20:29:24.540186 2339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:24.539717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:46339 (local address 127.0.0.1:56400)
0929 20:29:24.539919 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:24.540023 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:46339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 20:29:24.581437 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 2035
I20260929 20:29:24.600973 2339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:24.600485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:46339 (local address 127.0.0.1:56408)
0929 20:29:24.600713 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:24.600814 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:46339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260929 20:29:24.602133 2339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:24.601742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:46339 (local address 127.0.0.1:56414)
0929 20:29:24.601922 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:24.601991 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:46339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 20:29:24.605997 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 2165
I20260929 20:29:24.609356 2307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:24.606808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39924 (local address 127.28.244.131:44483)
0929 20:29:24.607020 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:24.607026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:24.607054 (+ 28us) server_negotiation.cc:407] Connection header received
0929 20:29:24.607222 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:24.607228 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:24.607302 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:24.607432 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:24.608572 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:24.608586 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:24.608591 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:29:24.608613 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:24.608656 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:29:24.608676 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:24.608819 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:24.608918 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 20:29:24.608998 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 20:29:24.610302 2287 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:29:24.683353 2339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:24.682848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:46339 (local address 127.0.0.1:56424)
0929 20:29:24.683105 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:24.683200 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:46339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260929 20:29:24.684670 2339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:24.684184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:46339 (local address 127.0.0.1:56440)
0929 20:29:24.684415 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:24.684527 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:46339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260929 20:29:24.729003 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 2165
I20260929 20:29:24.750115 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 1814
I20260929 20:29:24.751866 1875 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:29:24.864748 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 1814
2026-09-29T20:29:24Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (3081 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 20:29:24.907205 29650 mini_oidc.cc:152] Starting JWKS server
I20260929 20:29:24.909936 29650 webserver.cc:460] Webserver started at https://127.0.0.1:44557/ using document root <none> and password file <none>
I20260929 20:29:24.910305 29650 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 20:29:24.911003 29650 webserver.cc:460] Webserver started at http://127.0.0.1:33151/ using document root <none> and password file <none>
2026-09-29T20:29:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:29:24Z Disabled control of system clock
I20260929 20:29:24.940518 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:36435
--webserver_interface=127.28.244.190
--webserver_port=0
--builtin_ntp_servers=127.28.244.148:44427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:36435
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44557/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790713721201454-29650-0/testchainca.cert with env {}
W20260929 20:29:25.286187 2348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:25.286520 2348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:25.286609 2348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:25.294327 2348 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:29:25.294459 2348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:25.294502 2348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:25.294551 2348 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:29:25.294600 2348 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 20:29:25.294664 2348 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 20:29:25.294703 2348 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44557/jwks.json
I20260929 20:29:25.305300 2348 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:44427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:36435
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:36435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44557/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790713721201454-29650-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:25.306684 2348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:25.308068 2348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:25.321022 2354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:25.321230 2353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:25.321305 2356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:25.322669 2348 server_base.cc:1034] running on GCE node
I20260929 20:29:25.323685 2348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:25.325525 2348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:25.326835 2348 hybrid_clock.cc:648] HybridClock initialized: now 1790713765326787 us; error 47 us; skew 500 ppm
I20260929 20:29:25.327319 2348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:29:25.334403 2348 webserver.cc:460] Webserver started at http://127.28.244.190:39157/ using document root <none> and password file <none>
I20260929 20:29:25.335371 2348 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:25.335548 2348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:25.335973 2348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:25.339228 2348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "e76d0af4e3df4602a21f99fd83ca7502"
format_stamp: "Formatted at 2026-09-29 20:29:25 on dist-test-slave-p29l"
server_key: "fcd4d82fb2b4a7903302cbca73fd4fda"
server_key_iv: "2388ffaff7eb480a37169d79319db284"
server_key_version: "encryptionkey@0"
I20260929 20:29:25.340277 2348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "e76d0af4e3df4602a21f99fd83ca7502"
format_stamp: "Formatted at 2026-09-29 20:29:25 on dist-test-slave-p29l"
server_key: "fcd4d82fb2b4a7903302cbca73fd4fda"
server_key_iv: "2388ffaff7eb480a37169d79319db284"
server_key_version: "encryptionkey@0"
I20260929 20:29:25.346079 2348 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:29:25.350358 2362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:25.352329 2348 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 20:29:25.352607 2348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "e76d0af4e3df4602a21f99fd83ca7502"
format_stamp: "Formatted at 2026-09-29 20:29:25 on dist-test-slave-p29l"
server_key: "fcd4d82fb2b4a7903302cbca73fd4fda"
server_key_iv: "2388ffaff7eb480a37169d79319db284"
server_key_version: "encryptionkey@0"
I20260929 20:29:25.352862 2348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:25.370631 2348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:25.371630 2348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:25.372059 2348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:25.393013 2348 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:36435
I20260929 20:29:25.393034 2413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:36435 every 8 connection(s)
I20260929 20:29:25.396027 2348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:29:25.400660 2414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:25.402328 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 2348
I20260929 20:29:25.402594 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:29:25.403223 29650 external_mini_cluster.cc:1428] Setting key d6fef205989e8dba1928e1e059d765f0
I20260929 20:29:25.413592 2414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e76d0af4e3df4602a21f99fd83ca7502: Bootstrap starting.
I20260929 20:29:25.417534 2414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e76d0af4e3df4602a21f99fd83ca7502: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:25.418603 2417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:25.405395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34146 (local address 127.28.244.190:36435)
0929 20:29:25.406553 (+ 1158us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:25.406566 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:25.406616 (+ 50us) server_negotiation.cc:407] Connection header received
0929 20:29:25.408466 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:25.408493 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:25.409009 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:25.409399 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:25.410107 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:25.411527 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:25.413514 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:25.414342 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:25.414504 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:25.415874 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:25.415885 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:25.415900 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:29:25.416488 (+ 588us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:25.416589 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:29:25.416617 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:25.416710 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:25.417008 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 20:29:25.417428 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":424,"threads_started":1}
I20260929 20:29:25.418715 2414 log.cc:824] T 00000000000000000000000000000000 P e76d0af4e3df4602a21f99fd83ca7502: Log is configured to *not* fsync() on all Append() calls
I20260929 20:29:25.422197 2414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e76d0af4e3df4602a21f99fd83ca7502: No bootstrap required, opened a new log
I20260929 20:29:25.423215 29650 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:29:25.431267 2414 raft_consensus.cc:348] T 00000000000000000000000000000000 P e76d0af4e3df4602a21f99fd83ca7502 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e76d0af4e3df4602a21f99fd83ca7502" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 36435 } }
I20260929 20:29:25.431660 2414 raft_consensus.cc:374] T 00000000000000000000000000000000 P e76d0af4e3df4602a21f99fd83ca7502 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:25.431741 2414 raft_consensus.cc:729] T 00000000000000000000000000000000 P e76d0af4e3df4602a21f99fd83ca7502 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e76d0af4e3df4602a21f99fd83ca7502, State: Initialized, Role: FOLLOWER
I20260929 20:29:25.432428 2414 consensus_queue.cc:260] T 00000000000000000000000000000000 P e76d0af4e3df4602a21f99fd83ca7502 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e76d0af4e3df4602a21f99fd83ca7502" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 36435 } }
I20260929 20:29:25.432695 2414 raft_consensus.cc:388] T 00000000000000000000000000000000 P e76d0af4e3df4602a21f99fd83ca7502 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:29:25.432850 2414 raft_consensus.cc:482] T 00000000000000000000000000000000 P e76d0af4e3df4602a21f99fd83ca7502 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:29:25.433071 2414 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e76d0af4e3df4602a21f99fd83ca7502 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:25.435477 2414 raft_consensus.cc:504] T 00000000000000000000000000000000 P e76d0af4e3df4602a21f99fd83ca7502 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e76d0af4e3df4602a21f99fd83ca7502" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 36435 } }
I20260929 20:29:25.436103 2414 leader_election.cc:302] T 00000000000000000000000000000000 P e76d0af4e3df4602a21f99fd83ca7502 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e76d0af4e3df4602a21f99fd83ca7502; no voters:
I20260929 20:29:25.436664 2414 leader_election.cc:288] T 00000000000000000000000000000000 P e76d0af4e3df4602a21f99fd83ca7502 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:29:25.436842 2419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e76d0af4e3df4602a21f99fd83ca7502 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:29:25.438081 2419 raft_consensus.cc:686] T 00000000000000000000000000000000 P e76d0af4e3df4602a21f99fd83ca7502 [term 1 LEADER]: Becoming Leader. State: Replica: e76d0af4e3df4602a21f99fd83ca7502, State: Running, Role: LEADER
I20260929 20:29:25.438638 2419 consensus_queue.cc:237] T 00000000000000000000000000000000 P e76d0af4e3df4602a21f99fd83ca7502 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e76d0af4e3df4602a21f99fd83ca7502" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 36435 } }
I20260929 20:29:25.439138 2414 sys_catalog.cc:564] T 00000000000000000000000000000000 P e76d0af4e3df4602a21f99fd83ca7502 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:29:25.442021 2420 sys_catalog.cc:455] T 00000000000000000000000000000000 P e76d0af4e3df4602a21f99fd83ca7502 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e76d0af4e3df4602a21f99fd83ca7502" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e76d0af4e3df4602a21f99fd83ca7502" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 36435 } } }
I20260929 20:29:25.442008 2421 sys_catalog.cc:455] T 00000000000000000000000000000000 P e76d0af4e3df4602a21f99fd83ca7502 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e76d0af4e3df4602a21f99fd83ca7502. Latest consensus state: current_term: 1 leader_uuid: "e76d0af4e3df4602a21f99fd83ca7502" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e76d0af4e3df4602a21f99fd83ca7502" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 36435 } } }
I20260929 20:29:25.442584 2420 sys_catalog.cc:458] T 00000000000000000000000000000000 P e76d0af4e3df4602a21f99fd83ca7502 [sys.catalog]: This master's current role is: LEADER
I20260929 20:29:25.442590 2421 sys_catalog.cc:458] T 00000000000000000000000000000000 P e76d0af4e3df4602a21f99fd83ca7502 [sys.catalog]: This master's current role is: LEADER
I20260929 20:29:25.446511 2427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:29:25.452708 2427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:29:25.462332 2427 catalog_manager.cc:1347] Generated new cluster ID: 969394c3862f43b48197a7e0e1ed371f
I20260929 20:29:25.462484 2427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:29:25.481230 2427 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:29:25.483268 2427 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:29:25.497870 2427 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e76d0af4e3df4602a21f99fd83ca7502: Generated new TSK 0
I20260929 20:29:25.499097 2427 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 20:29:25.546056 2444 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 20:29:25.546262 2444 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 20:29:25.547449 2417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:25.522880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34152 (local address 127.28.244.190:36435)
0929 20:29:25.523116 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:25.523123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:25.523645 (+ 522us) server_negotiation.cc:407] Connection header received
0929 20:29:25.523997 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:25.524005 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:25.524115 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:25.524277 (+ 162us) server_negotiation.cc:226] Negotiated authn=JWT
0929 20:29:25.524882 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:25.526095 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:25.527865 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:25.528585 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:25.528722 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:25.529332 (+ 610us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 20:29:25.546716 (+ 17384us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44557/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 20:29:25.547135 (+ 419us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34152: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44557/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":84}
W20260929 20:29:25.547578 2417 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34152: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44557/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 20:29:25.549116 2443 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:29:25.522790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:36435 (local address 127.0.0.1:34152)
0929 20:29:25.523525 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:25.523542 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:25.523707 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:25.524332 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:25.524336 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:25.524346 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0929 20:29:25.524715 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:25.524724 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:25.526340 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:25.526346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:25.527666 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:25.527681 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:25.528794 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:25.528801 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:25.528948 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:25.529027 (+ 79us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 20:29:25.547090 (+ 18063us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44557/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 20:29:25.547439 (+ 349us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.244.190:36435: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44557/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":383,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.28.244.190:36435: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44557/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 20:29:25.550899 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 2348
I20260929 20:29:25.552348 2409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:29:25.650715 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 2348
2026-09-29T20:29:25Z chronyd exiting
I20260929 20:29:25.702572 29650 test_util.cc:175] -----------------------------------------------
I20260929 20:29:25.702760 29650 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790713721201454-29650-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (800 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 20:29:25.706634 29650 mini_oidc.cc:152] Starting JWKS server
I20260929 20:29:25.708606 29650 webserver.cc:460] Webserver started at https://127.0.0.1:38735/ using document root <none> and password file <none>
I20260929 20:29:25.708865 29650 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 20:29:25.709561 29650 webserver.cc:460] Webserver started at http://127.0.0.1:40119/ using document root <none> and password file <none>
2026-09-29T20:29:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:29:25Z Disabled control of system clock
I20260929 20:29:25.739353 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:34289
--webserver_interface=127.28.244.190
--webserver_port=0
--builtin_ntp_servers=127.28.244.148:37343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:34289
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38735/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790713721201454-29650-0/testchainca.cert with env {}
W20260929 20:29:26.083462 2451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:26.083786 2451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:26.083961 2451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:26.091554 2451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:29:26.091681 2451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:26.091722 2451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:26.091773 2451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:29:26.091885 2451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 20:29:26.091962 2451 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 20:29:26.091995 2451 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38735/jwks.json
I20260929 20:29:26.102661 2451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:37343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:34289
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:34289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38735/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790713721201454-29650-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:26.104143 2451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:26.105499 2451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:26.119104 2456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:26.119191 2457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:26.119148 2459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:26.120313 2451 server_base.cc:1034] running on GCE node
I20260929 20:29:26.121373 2451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:26.123063 2451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:26.124321 2451 hybrid_clock.cc:648] HybridClock initialized: now 1790713766124221 us; error 95 us; skew 500 ppm
I20260929 20:29:26.124747 2451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:29:26.127334 2451 webserver.cc:460] Webserver started at http://127.28.244.190:39899/ using document root <none> and password file <none>
I20260929 20:29:26.128206 2451 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:26.128345 2451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:26.128693 2451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:26.131912 2451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "ed107f4ddf8a442183249b02595c3100"
format_stamp: "Formatted at 2026-09-29 20:29:26 on dist-test-slave-p29l"
server_key: "650166ffb79312c58b606c520836ac15"
server_key_iv: "70805917e6765bd594a3e939e65a6c55"
server_key_version: "encryptionkey@0"
I20260929 20:29:26.132843 2451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "ed107f4ddf8a442183249b02595c3100"
format_stamp: "Formatted at 2026-09-29 20:29:26 on dist-test-slave-p29l"
server_key: "650166ffb79312c58b606c520836ac15"
server_key_iv: "70805917e6765bd594a3e939e65a6c55"
server_key_version: "encryptionkey@0"
I20260929 20:29:26.138871 2451 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 20:29:26.143046 2465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:26.144749 2451 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:29:26.144974 2451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "ed107f4ddf8a442183249b02595c3100"
format_stamp: "Formatted at 2026-09-29 20:29:26 on dist-test-slave-p29l"
server_key: "650166ffb79312c58b606c520836ac15"
server_key_iv: "70805917e6765bd594a3e939e65a6c55"
server_key_version: "encryptionkey@0"
I20260929 20:29:26.145278 2451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:26.166095 2451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:26.166978 2451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:26.167251 2451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:26.186889 2451 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:34289
I20260929 20:29:26.186923 2516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:34289 every 8 connection(s)
I20260929 20:29:26.188395 2451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:29:26.191246 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 2451
I20260929 20:29:26.191453 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:29:26.192356 29650 external_mini_cluster.cc:1428] Setting key 4f2b4cd59db938efa14a4678221c863f
I20260929 20:29:26.193482 2517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:26.207062 2520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:26.194379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36652 (local address 127.28.244.190:34289)
0929 20:29:26.195456 (+ 1077us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:26.195468 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:26.195511 (+ 43us) server_negotiation.cc:407] Connection header received
0929 20:29:26.197113 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:26.197140 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:26.197687 (+ 547us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:26.198109 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:26.198747 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:26.200270 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:26.202148 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:26.202876 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:26.203047 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:26.204486 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:26.204499 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:26.204516 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:29:26.205159 (+ 643us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:26.205248 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:29:26.205272 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:26.205368 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:26.205625 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 20:29:26.206052 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":396,"threads_started":1}
I20260929 20:29:26.208329 2517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed107f4ddf8a442183249b02595c3100: Bootstrap starting.
I20260929 20:29:26.211293 29650 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:29:26.212020 2517 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed107f4ddf8a442183249b02595c3100: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:26.213390 2517 log.cc:824] T 00000000000000000000000000000000 P ed107f4ddf8a442183249b02595c3100: Log is configured to *not* fsync() on all Append() calls
I20260929 20:29:26.217069 2517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed107f4ddf8a442183249b02595c3100: No bootstrap required, opened a new log
I20260929 20:29:26.224558 2517 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed107f4ddf8a442183249b02595c3100 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed107f4ddf8a442183249b02595c3100" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34289 } }
I20260929 20:29:26.225021 2517 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed107f4ddf8a442183249b02595c3100 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:26.225132 2517 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed107f4ddf8a442183249b02595c3100 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed107f4ddf8a442183249b02595c3100, State: Initialized, Role: FOLLOWER
I20260929 20:29:26.225804 2517 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed107f4ddf8a442183249b02595c3100 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed107f4ddf8a442183249b02595c3100" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34289 } }
I20260929 20:29:26.226065 2517 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed107f4ddf8a442183249b02595c3100 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:29:26.226207 2517 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed107f4ddf8a442183249b02595c3100 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:29:26.226480 2517 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed107f4ddf8a442183249b02595c3100 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:26.228884 2517 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed107f4ddf8a442183249b02595c3100 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed107f4ddf8a442183249b02595c3100" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34289 } }
I20260929 20:29:26.229476 2517 leader_election.cc:302] T 00000000000000000000000000000000 P ed107f4ddf8a442183249b02595c3100 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed107f4ddf8a442183249b02595c3100; no voters:
I20260929 20:29:26.230218 2517 leader_election.cc:288] T 00000000000000000000000000000000 P ed107f4ddf8a442183249b02595c3100 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:29:26.230417 2522 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed107f4ddf8a442183249b02595c3100 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:29:26.231191 2522 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed107f4ddf8a442183249b02595c3100 [term 1 LEADER]: Becoming Leader. State: Replica: ed107f4ddf8a442183249b02595c3100, State: Running, Role: LEADER
I20260929 20:29:26.231734 2522 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed107f4ddf8a442183249b02595c3100 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed107f4ddf8a442183249b02595c3100" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34289 } }
I20260929 20:29:26.232679 2517 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed107f4ddf8a442183249b02595c3100 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:29:26.235028 2523 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed107f4ddf8a442183249b02595c3100 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed107f4ddf8a442183249b02595c3100" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed107f4ddf8a442183249b02595c3100" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34289 } } }
I20260929 20:29:26.235533 2523 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed107f4ddf8a442183249b02595c3100 [sys.catalog]: This master's current role is: LEADER
I20260929 20:29:26.235356 2524 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed107f4ddf8a442183249b02595c3100 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed107f4ddf8a442183249b02595c3100. Latest consensus state: current_term: 1 leader_uuid: "ed107f4ddf8a442183249b02595c3100" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed107f4ddf8a442183249b02595c3100" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34289 } } }
I20260929 20:29:26.235695 2524 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed107f4ddf8a442183249b02595c3100 [sys.catalog]: This master's current role is: LEADER
I20260929 20:29:26.237991 2531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:29:26.243661 2531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:29:26.254096 2531 catalog_manager.cc:1347] Generated new cluster ID: f73da9b4c03a4a9fbe7960018694d386
I20260929 20:29:26.254252 2531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:29:26.266183 2531 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:29:26.268154 2531 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:29:26.279114 2531 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed107f4ddf8a442183249b02595c3100: Generated new TSK 0
I20260929 20:29:26.280131 2531 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 20:29:26.334304 2547 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 20:29:26.334448 2547 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 20:29:26.335768 2520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:26.307028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36654 (local address 127.28.244.190:34289)
0929 20:29:26.307247 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:26.307254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:26.307744 (+ 490us) server_negotiation.cc:407] Connection header received
0929 20:29:26.308079 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:26.308085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:26.308184 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:26.308370 (+ 186us) server_negotiation.cc:226] Negotiated authn=JWT
0929 20:29:26.308926 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:26.310129 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:26.311717 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:26.312431 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:26.312608 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:26.313174 (+ 566us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 20:29:26.334991 (+ 21817us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38735/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 20:29:26.335430 (+ 439us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36654: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38735/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 20:29:26.335952 2520 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36654: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38735/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260929 20:29:26.336483 2546 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:29:26.306855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:34289 (local address 127.0.0.1:36654)
0929 20:29:26.307629 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:26.307647 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:26.307788 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:26.308410 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:26.308416 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:26.308430 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0929 20:29:26.308739 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:26.308749 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:26.310284 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:26.310288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:26.311530 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:26.311544 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:26.312647 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:26.312654 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:26.312824 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:26.312866 (+ 42us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 20:29:26.335264 (+ 22398us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38735/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 20:29:26.335559 (+ 295us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.244.190:34289: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38735/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":454,"threads_started":1}
I20260929 20:29:26.337484 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 2451
I20260929 20:29:26.338961 2512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:29:26.434609 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 2451
2026-09-29T20:29:26Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (784 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 20:29:26.491567 29650 mini_oidc.cc:152] Starting JWKS server
I20260929 20:29:26.494906 29650 webserver.cc:460] Webserver started at https://127.0.0.1:36505/ using document root <none> and password file <none>
I20260929 20:29:26.495309 29650 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 20:29:26.496294 29650 webserver.cc:460] Webserver started at http://127.0.0.1:34551/ using document root <none> and password file <none>
2026-09-29T20:29:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:29:26Z Disabled control of system clock
I20260929 20:29:26.525382 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:33463
--webserver_interface=127.28.244.190
--webserver_port=0
--builtin_ntp_servers=127.28.244.148:39527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:33463
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36505/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 20:29:26.862768 2554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:26.863174 2554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:26.863338 2554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:26.871007 2554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:29:26.871191 2554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:26.871281 2554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:26.871397 2554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:29:26.871477 2554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 20:29:26.871568 2554 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 20:29:26.871644 2554 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36505/jwks.json
I20260929 20:29:26.881628 2554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:39527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:33463
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:33463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36505/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:26.883080 2554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:26.885134 2554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:26.898303 2562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:26.898375 2560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:26.898679 2559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:26.899880 2554 server_base.cc:1034] running on GCE node
I20260929 20:29:26.900810 2554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:26.902439 2554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:26.903709 2554 hybrid_clock.cc:648] HybridClock initialized: now 1790713766903670 us; error 64 us; skew 500 ppm
I20260929 20:29:26.904139 2554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:29:26.906718 2554 webserver.cc:460] Webserver started at http://127.28.244.190:40519/ using document root <none> and password file <none>
I20260929 20:29:26.907526 2554 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:26.907673 2554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:26.908092 2554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:26.911309 2554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "4db287cdc95a4e549228c98be0175da2"
format_stamp: "Formatted at 2026-09-29 20:29:26 on dist-test-slave-p29l"
server_key: "10d4522ce43d606b0de53b2c29b42b58"
server_key_iv: "a3b553ec4bea1cbb79717b7d8a009275"
server_key_version: "encryptionkey@0"
I20260929 20:29:26.912366 2554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "4db287cdc95a4e549228c98be0175da2"
format_stamp: "Formatted at 2026-09-29 20:29:26 on dist-test-slave-p29l"
server_key: "10d4522ce43d606b0de53b2c29b42b58"
server_key_iv: "a3b553ec4bea1cbb79717b7d8a009275"
server_key_version: "encryptionkey@0"
I20260929 20:29:26.918251 2554 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:29:26.922070 2568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:26.923661 2554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 20:29:26.923918 2554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "4db287cdc95a4e549228c98be0175da2"
format_stamp: "Formatted at 2026-09-29 20:29:26 on dist-test-slave-p29l"
server_key: "10d4522ce43d606b0de53b2c29b42b58"
server_key_iv: "a3b553ec4bea1cbb79717b7d8a009275"
server_key_version: "encryptionkey@0"
I20260929 20:29:26.924180 2554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:26.946727 2554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:26.947659 2554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:26.948069 2554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:26.967314 2554 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:33463
I20260929 20:29:26.967320 2619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:33463 every 8 connection(s)
I20260929 20:29:26.968866 2554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:29:26.973101 2620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:26.976325 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 2554
I20260929 20:29:26.976541 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:29:26.977366 29650 external_mini_cluster.cc:1428] Setting key 3afe7806ce174a4127cf1106039e0172
I20260929 20:29:26.987860 2620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4db287cdc95a4e549228c98be0175da2: Bootstrap starting.
I20260929 20:29:26.991897 2620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4db287cdc95a4e549228c98be0175da2: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:26.993219 2620 log.cc:824] T 00000000000000000000000000000000 P 4db287cdc95a4e549228c98be0175da2: Log is configured to *not* fsync() on all Append() calls
I20260929 20:29:26.993732 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:26.979536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58968 (local address 127.28.244.190:33463)
0929 20:29:26.980771 (+ 1235us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:26.980786 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:26.980852 (+ 66us) server_negotiation.cc:407] Connection header received
0929 20:29:26.983136 (+ 2284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:26.983174 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:26.983870 (+ 696us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:26.984355 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:26.984965 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:26.986961 (+ 1996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:26.988487 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:26.989532 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:26.989766 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:26.990900 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:26.990913 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:26.990934 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:29:26.991658 (+ 724us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:26.991792 (+ 134us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:29:26.991846 (+ 54us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:26.991988 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:26.992259 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 20:29:26.992661 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":474,"threads_started":1}
I20260929 20:29:26.996402 2620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4db287cdc95a4e549228c98be0175da2: No bootstrap required, opened a new log
I20260929 20:29:26.997982 29650 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:29:27.005139 2620 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4db287cdc95a4e549228c98be0175da2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4db287cdc95a4e549228c98be0175da2" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 33463 } }
I20260929 20:29:27.005533 2620 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4db287cdc95a4e549228c98be0175da2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:27.005609 2620 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4db287cdc95a4e549228c98be0175da2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4db287cdc95a4e549228c98be0175da2, State: Initialized, Role: FOLLOWER
I20260929 20:29:27.006311 2620 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4db287cdc95a4e549228c98be0175da2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4db287cdc95a4e549228c98be0175da2" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 33463 } }
I20260929 20:29:27.006635 2620 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4db287cdc95a4e549228c98be0175da2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:29:27.006794 2620 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4db287cdc95a4e549228c98be0175da2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:29:27.006968 2620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4db287cdc95a4e549228c98be0175da2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:27.009289 2620 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4db287cdc95a4e549228c98be0175da2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4db287cdc95a4e549228c98be0175da2" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 33463 } }
I20260929 20:29:27.009837 2620 leader_election.cc:302] T 00000000000000000000000000000000 P 4db287cdc95a4e549228c98be0175da2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4db287cdc95a4e549228c98be0175da2; no voters:
I20260929 20:29:27.010594 2620 leader_election.cc:288] T 00000000000000000000000000000000 P 4db287cdc95a4e549228c98be0175da2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:29:27.010773 2625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4db287cdc95a4e549228c98be0175da2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:29:27.011579 2625 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4db287cdc95a4e549228c98be0175da2 [term 1 LEADER]: Becoming Leader. State: Replica: 4db287cdc95a4e549228c98be0175da2, State: Running, Role: LEADER
I20260929 20:29:27.012197 2625 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4db287cdc95a4e549228c98be0175da2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4db287cdc95a4e549228c98be0175da2" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 33463 } }
I20260929 20:29:27.012884 2620 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4db287cdc95a4e549228c98be0175da2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:29:27.015259 2627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4db287cdc95a4e549228c98be0175da2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4db287cdc95a4e549228c98be0175da2. Latest consensus state: current_term: 1 leader_uuid: "4db287cdc95a4e549228c98be0175da2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4db287cdc95a4e549228c98be0175da2" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 33463 } } }
I20260929 20:29:27.015304 2626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4db287cdc95a4e549228c98be0175da2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4db287cdc95a4e549228c98be0175da2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4db287cdc95a4e549228c98be0175da2" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 33463 } } }
I20260929 20:29:27.015765 2626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4db287cdc95a4e549228c98be0175da2 [sys.catalog]: This master's current role is: LEADER
I20260929 20:29:27.015663 2627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4db287cdc95a4e549228c98be0175da2 [sys.catalog]: This master's current role is: LEADER
I20260929 20:29:27.018358 2634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:29:27.023906 2634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:29:27.033444 2634 catalog_manager.cc:1347] Generated new cluster ID: 89e99edf7b3e48ebad2cf4ff263ff9c0
I20260929 20:29:27.033577 2634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:29:27.052404 2634 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:29:27.053931 2634 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:29:27.062861 2634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4db287cdc95a4e549228c98be0175da2: Generated new TSK 0
I20260929 20:29:27.063934 2634 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:29:27.086158 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:27.077422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58972 (local address 127.28.244.190:33463)
0929 20:29:27.077627 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:27.077635 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:27.077996 (+ 361us) server_negotiation.cc:407] Connection header received
0929 20:29:27.078219 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:27.078225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:27.078327 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:27.078524 (+ 197us) server_negotiation.cc:226] Negotiated authn=JWT
0929 20:29:27.079004 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:27.080198 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:27.081800 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:27.082489 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:27.082627 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:27.083207 (+ 580us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 20:29:27.085424 (+ 2217us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36505/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 20:29:27.085832 (+ 408us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58972: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36505/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":80}
W20260929 20:29:27.086301 2623 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58972: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36505/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 20:29:27.086333 2649 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:29:27.077256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:33463 (local address 127.0.0.1:58972)
0929 20:29:27.077886 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:27.077903 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:27.078037 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:27.078548 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:27.078552 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:27.078561 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0929 20:29:27.078830 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:27.078838 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:27.080385 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:27.080390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:27.081633 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:27.081648 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:27.082708 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:27.082713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:27.082865 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:27.082906 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 20:29:27.085670 (+ 2764us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36505/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 20:29:27.085926 (+ 256us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.244.190:33463: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36505/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":368,"threads_started":1}
I20260929 20:29:27.087044 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 2554
I20260929 20:29:27.088382 2615 generic_service.cc:196] Checking for leaks (request via RPC)
E20260929 20:29:27.098763 2650 webserver.cc:533] Webserver: sslize failed:
E20260929 20:29:27.098874 2650 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 20:29:27.180747 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 2554
2026-09-29T20:29:27Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (745 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T20:29:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:29:27Z Disabled control of system clock
I20260929 20:29:27.265064 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:44009
--webserver_interface=127.28.244.190
--webserver_port=0
--builtin_ntp_servers=127.28.244.148:34641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:44009
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790713721201454-29650-0/testchainca.cert with env {}
W20260929 20:29:27.607975 2655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:27.608314 2655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:27.608474 2655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:27.616107 2655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:29:27.616226 2655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:27.616298 2655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:27.616523 2655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:29:27.616676 2655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 20:29:27.616797 2655 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 20:29:27.616907 2655 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 20:29:27.627564 2655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:34641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:44009
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:44009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790713721201454-29650-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:27.628984 2655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:27.630519 2655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:27.643432 2660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:27.643404 2663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:27.643366 2661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:27.644438 2655 server_base.cc:1034] running on GCE node
I20260929 20:29:27.645324 2655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:27.646886 2655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:27.648178 2655 hybrid_clock.cc:648] HybridClock initialized: now 1790713767648110 us; error 66 us; skew 500 ppm
I20260929 20:29:27.648618 2655 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:29:27.651381 2655 webserver.cc:460] Webserver started at http://127.28.244.190:44133/ using document root <none> and password file <none>
I20260929 20:29:27.652335 2655 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:27.652490 2655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:27.652832 2655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:27.656230 2655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "92d7fe7723bd4302929143b6413b7b00"
format_stamp: "Formatted at 2026-09-29 20:29:27 on dist-test-slave-p29l"
server_key: "05edec69dd50c9b709fed6023f6eec7a"
server_key_iv: "0b69fceb5e54f4a654f02b68d659267b"
server_key_version: "encryptionkey@0"
I20260929 20:29:27.657142 2655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "92d7fe7723bd4302929143b6413b7b00"
format_stamp: "Formatted at 2026-09-29 20:29:27 on dist-test-slave-p29l"
server_key: "05edec69dd50c9b709fed6023f6eec7a"
server_key_iv: "0b69fceb5e54f4a654f02b68d659267b"
server_key_version: "encryptionkey@0"
I20260929 20:29:27.663445 2655 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260929 20:29:27.667357 2669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:27.668931 2655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:29:27.669143 2655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "92d7fe7723bd4302929143b6413b7b00"
format_stamp: "Formatted at 2026-09-29 20:29:27 on dist-test-slave-p29l"
server_key: "05edec69dd50c9b709fed6023f6eec7a"
server_key_iv: "0b69fceb5e54f4a654f02b68d659267b"
server_key_version: "encryptionkey@0"
I20260929 20:29:27.669406 2655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:27.689998 2655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:27.690940 2655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:27.691278 2655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:27.711007 2655 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:44009
I20260929 20:29:27.711035 2720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:44009 every 8 connection(s)
I20260929 20:29:27.712478 2655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:29:27.716741 2721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:27.716751 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 2655
I20260929 20:29:27.716953 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:29:27.717617 29650 external_mini_cluster.cc:1428] Setting key 2fc7c643f77ae39d23d4fc281544c650
I20260929 20:29:27.731928 2721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92d7fe7723bd4302929143b6413b7b00: Bootstrap starting.
I20260929 20:29:27.734481 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:27.719605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40676 (local address 127.28.244.190:44009)
0929 20:29:27.720825 (+ 1220us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:27.720847 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:27.720900 (+ 53us) server_negotiation.cc:407] Connection header received
0929 20:29:27.723608 (+ 2708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:27.723640 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:27.724378 (+ 738us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:27.724894 (+ 516us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:27.725452 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:27.727402 (+ 1950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:27.728870 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:27.729996 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:27.730182 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:27.731393 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:27.731409 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:27.731429 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:29:27.732185 (+ 756us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:27.732305 (+ 120us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:29:27.732336 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:27.732459 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:27.732717 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 20:29:27.733079 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":427,"threads_started":1}
I20260929 20:29:27.736235 2721 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92d7fe7723bd4302929143b6413b7b00: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:27.737461 2721 log.cc:824] T 00000000000000000000000000000000 P 92d7fe7723bd4302929143b6413b7b00: Log is configured to *not* fsync() on all Append() calls
I20260929 20:29:27.739154 29650 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:29:27.741055 2721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92d7fe7723bd4302929143b6413b7b00: No bootstrap required, opened a new log
I20260929 20:29:27.749116 2721 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92d7fe7723bd4302929143b6413b7b00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92d7fe7723bd4302929143b6413b7b00" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 44009 } }
I20260929 20:29:27.749635 2721 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92d7fe7723bd4302929143b6413b7b00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:27.749747 2721 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92d7fe7723bd4302929143b6413b7b00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92d7fe7723bd4302929143b6413b7b00, State: Initialized, Role: FOLLOWER
I20260929 20:29:27.750394 2721 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92d7fe7723bd4302929143b6413b7b00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92d7fe7723bd4302929143b6413b7b00" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 44009 } }
I20260929 20:29:27.750622 2721 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92d7fe7723bd4302929143b6413b7b00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:29:27.750769 2721 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92d7fe7723bd4302929143b6413b7b00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:29:27.750946 2721 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92d7fe7723bd4302929143b6413b7b00 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:27.753203 2721 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92d7fe7723bd4302929143b6413b7b00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92d7fe7723bd4302929143b6413b7b00" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 44009 } }
I20260929 20:29:27.753773 2721 leader_election.cc:302] T 00000000000000000000000000000000 P 92d7fe7723bd4302929143b6413b7b00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92d7fe7723bd4302929143b6413b7b00; no voters:
I20260929 20:29:27.754449 2721 leader_election.cc:288] T 00000000000000000000000000000000 P 92d7fe7723bd4302929143b6413b7b00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:29:27.754599 2726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92d7fe7723bd4302929143b6413b7b00 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:29:27.756181 2726 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92d7fe7723bd4302929143b6413b7b00 [term 1 LEADER]: Becoming Leader. State: Replica: 92d7fe7723bd4302929143b6413b7b00, State: Running, Role: LEADER
I20260929 20:29:27.756728 2721 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92d7fe7723bd4302929143b6413b7b00 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:29:27.756773 2726 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92d7fe7723bd4302929143b6413b7b00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92d7fe7723bd4302929143b6413b7b00" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 44009 } }
I20260929 20:29:27.759972 2727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92d7fe7723bd4302929143b6413b7b00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92d7fe7723bd4302929143b6413b7b00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92d7fe7723bd4302929143b6413b7b00" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 44009 } } }
I20260929 20:29:27.760396 2727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92d7fe7723bd4302929143b6413b7b00 [sys.catalog]: This master's current role is: LEADER
I20260929 20:29:27.759968 2728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92d7fe7723bd4302929143b6413b7b00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92d7fe7723bd4302929143b6413b7b00. Latest consensus state: current_term: 1 leader_uuid: "92d7fe7723bd4302929143b6413b7b00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92d7fe7723bd4302929143b6413b7b00" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 44009 } } }
I20260929 20:29:27.760936 2728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92d7fe7723bd4302929143b6413b7b00 [sys.catalog]: This master's current role is: LEADER
I20260929 20:29:27.765044 2734 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:29:27.771044 2734 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:29:27.781352 2734 catalog_manager.cc:1347] Generated new cluster ID: adab2b1d776f4f53b59fea0212e3cf23
I20260929 20:29:27.781490 2734 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:29:27.799507 2734 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:29:27.800949 2734 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:29:27.811568 2734 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92d7fe7723bd4302929143b6413b7b00: Generated new TSK 0
I20260929 20:29:27.812569 2734 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:29:28.348167 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:27.827777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40688 (local address 127.28.244.190:44009)
0929 20:29:27.828082 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:27.828093 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:27.828413 (+ 320us) server_negotiation.cc:407] Connection header received
0929 20:29:27.828683 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:27.828687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:27.828763 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:27.828942 (+ 179us) server_negotiation.cc:226] Negotiated authn=JWT
0929 20:29:27.829435 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:27.830177 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:27.831650 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:27.832183 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:27.832341 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:27.832862 (+ 521us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 20:29:28.347326 (+514464us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 20:29:28.347850 (+ 524us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40688: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":152}
W20260929 20:29:28.348321 2724 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40688: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 20:29:28.348423 2750 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:29:27.827595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:44009 (local address 127.0.0.1:40688)
0929 20:29:27.828286 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:27.828310 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:27.828496 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:27.828969 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:27.828974 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:27.828987 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0929 20:29:27.829259 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:27.829268 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:27.830360 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:27.830365 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:27.831494 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:27.831505 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:27.832394 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:27.832402 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:27.832544 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:27.832582 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 20:29:28.347663 (+515081us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 20:29:28.348020 (+ 357us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.244.190:44009: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":391,"threads_started":1}
I20260929 20:29:28.349774 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 2655
I20260929 20:29:28.351042 2716 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:29:28.455195 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 2655
2026-09-29T20:29:28Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1256 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (431 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1521 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:29:30 dist-test-slave-p29l krb5kdc[2766](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:29:30 dist-test-slave-p29l krb5kdc[2766](info): set up 2 sockets
Sep 29 20:29:30 dist-test-slave-p29l krb5kdc[2766](info): commencing operation
krb5kdc: starting...
W20260929 20:29:31.504146 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.005s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:29:31 dist-test-slave-p29l krb5kdc[2766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:29:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:29:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.244.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
I20260929 20:29:31.728434 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:41093
--webserver_interface=127.28.244.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:38615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:41093
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:29:32.062544 2779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:32.062942 2779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:32.063102 2779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:32.070802 2779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:29:32.070995 2779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:32.071117 2779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:32.071215 2779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:29:32.071302 2779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:29:32.081681 2779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:38615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:41093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:41093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:32.083189 2779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:32.084630 2779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:32.097162 2784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:32.097450 2785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:32.097409 2787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:32.099021 2779 server_base.cc:1034] running on GCE node
I20260929 20:29:32.099972 2779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:32.101504 2779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:32.102730 2779 hybrid_clock.cc:648] HybridClock initialized: now 1790713772102666 us; error 56 us; skew 500 ppm
I20260929 20:29:32.103128 2779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:32 dist-test-slave-p29l krb5kdc[2766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713772, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:32.106729 2779 init.cc:377] Logged in from keytab as kudu/127.28.244.190@KRBTEST.COM (short username kudu)
I20260929 20:29:32.108673 2779 webserver.cc:460] Webserver started at http://127.28.244.190:39459/ using document root <none> and password file <none>
I20260929 20:29:32.109512 2779 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:32.109665 2779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:32.110071 2779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:32.113519 2779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "4ca08d026915433ab644747b21abf014"
format_stamp: "Formatted at 2026-09-29 20:29:32 on dist-test-slave-p29l"
server_key: "1568107becc9197fd5d40f5c29ef2672"
server_key_iv: "235ee9931310659020ad67245980a7d6"
server_key_version: "encryptionkey@0"
I20260929 20:29:32.114713 2779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "4ca08d026915433ab644747b21abf014"
format_stamp: "Formatted at 2026-09-29 20:29:32 on dist-test-slave-p29l"
server_key: "1568107becc9197fd5d40f5c29ef2672"
server_key_iv: "235ee9931310659020ad67245980a7d6"
server_key_version: "encryptionkey@0"
I20260929 20:29:32.121001 2779 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 20:29:32.125068 2794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:32.126706 2779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:29:32.126962 2779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "4ca08d026915433ab644747b21abf014"
format_stamp: "Formatted at 2026-09-29 20:29:32 on dist-test-slave-p29l"
server_key: "1568107becc9197fd5d40f5c29ef2672"
server_key_iv: "235ee9931310659020ad67245980a7d6"
server_key_version: "encryptionkey@0"
I20260929 20:29:32.127202 2779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:32.150537 2779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:32.155974 2779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:32.156253 2779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:32.176326 2779 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:41093
I20260929 20:29:32.176357 2845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:41093 every 8 connection(s)
I20260929 20:29:32.177836 2779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:29:32.180122 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 2779
I20260929 20:29:32.180326 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:29:32.181038 29650 external_mini_cluster.cc:1428] Setting key 3f423a51c6e33355fffe257603c50c58
I20260929 20:29:32.182489 2846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 20:29:32 dist-test-slave-p29l krb5kdc[2766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:29:32.197465 2846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ca08d026915433ab644747b21abf014: Bootstrap starting.
I20260929 20:29:32.201059 2846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ca08d026915433ab644747b21abf014: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:32.202303 2846 log.cc:824] T 00000000000000000000000000000000 P 4ca08d026915433ab644747b21abf014: Log is configured to *not* fsync() on all Append() calls
I20260929 20:29:32.205595 2846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ca08d026915433ab644747b21abf014: No bootstrap required, opened a new log
I20260929 20:29:32.206460 2849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:32.183070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57070 (local address 127.28.244.190:41093)
0929 20:29:32.184009 (+ 939us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:32.184024 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:32.184077 (+ 53us) server_negotiation.cc:407] Connection header received
0929 20:29:32.186155 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:32.186186 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:32.186771 (+ 585us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:32.187269 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:32.188883 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:32.191076 (+ 2193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:32.192639 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:32.193749 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:32.193968 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:32.198395 (+ 4427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:32.198406 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:32.198431 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:32.198476 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:32.201763 (+ 3287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:32.202699 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:32.202706 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:32.202714 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:32.202880 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:32.203349 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:32.203353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:32.203355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:32.204022 (+ 667us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:32.204326 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:32.204826 (+ 500us) server_negotiation.cc:299] Negotiation successful
0929 20:29:32.205290 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":382,"threads_started":1}
I20260929 20:29:32.214346 2846 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ca08d026915433ab644747b21abf014 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca08d026915433ab644747b21abf014" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 41093 } }
I20260929 20:29:32.214799 2846 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ca08d026915433ab644747b21abf014 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:32.214921 2846 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ca08d026915433ab644747b21abf014 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ca08d026915433ab644747b21abf014, State: Initialized, Role: FOLLOWER
I20260929 20:29:32.215615 2846 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ca08d026915433ab644747b21abf014 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca08d026915433ab644747b21abf014" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 41093 } }
I20260929 20:29:32.215917 2846 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ca08d026915433ab644747b21abf014 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:29:32.216084 2846 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ca08d026915433ab644747b21abf014 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:29:32.216298 2846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ca08d026915433ab644747b21abf014 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:32.218757 2846 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ca08d026915433ab644747b21abf014 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca08d026915433ab644747b21abf014" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 41093 } }
I20260929 20:29:32.219297 2846 leader_election.cc:302] T 00000000000000000000000000000000 P 4ca08d026915433ab644747b21abf014 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ca08d026915433ab644747b21abf014; no voters:
I20260929 20:29:32.220031 2846 leader_election.cc:288] T 00000000000000000000000000000000 P 4ca08d026915433ab644747b21abf014 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:29:32.220206 2851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ca08d026915433ab644747b21abf014 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:29:32.221921 2851 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ca08d026915433ab644747b21abf014 [term 1 LEADER]: Becoming Leader. State: Replica: 4ca08d026915433ab644747b21abf014, State: Running, Role: LEADER
I20260929 20:29:32.222210 2846 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ca08d026915433ab644747b21abf014 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:29:32.222484 2851 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ca08d026915433ab644747b21abf014 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca08d026915433ab644747b21abf014" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 41093 } }
I20260929 20:29:32.225852 2853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ca08d026915433ab644747b21abf014 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ca08d026915433ab644747b21abf014. Latest consensus state: current_term: 1 leader_uuid: "4ca08d026915433ab644747b21abf014" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca08d026915433ab644747b21abf014" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 41093 } } }
I20260929 20:29:32.225824 2852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ca08d026915433ab644747b21abf014 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ca08d026915433ab644747b21abf014" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca08d026915433ab644747b21abf014" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 41093 } } }
I20260929 20:29:32.226298 2853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ca08d026915433ab644747b21abf014 [sys.catalog]: This master's current role is: LEADER
I20260929 20:29:32.226300 2852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ca08d026915433ab644747b21abf014 [sys.catalog]: This master's current role is: LEADER
I20260929 20:29:32.228698 2860 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:29:32.233741 2860 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:29:32.251096 2860 catalog_manager.cc:1347] Generated new cluster ID: 3603f3ffc55e4a9d805a0aa582e22188
I20260929 20:29:32.251261 2860 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:29:32.264007 2860 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:29:32.265396 2860 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:29:32.277259 2860 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ca08d026915433ab644747b21abf014: Generated new TSK 0
I20260929 20:29:32.278234 2860 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.244.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab.
Entry for principal kudu/127.28.244.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab.
I20260929 20:29:32.341277 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.129:0
--local_ip_for_outbound_sockets=127.28.244.129
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:41093
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:38615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:29:32.691682 2872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:32.692050 2872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:32.692240 2872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:32.699668 2872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:32.699784 2872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:32.699966 2872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.129
I20260929 20:29:32.710515 2872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:38615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:41093
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.129
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:32.711890 2872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:32.713214 2872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:32.726505 2878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:32.726541 2877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:32.726562 2880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:32.727190 2872 server_base.cc:1034] running on GCE node
I20260929 20:29:32.728027 2872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:32.729033 2872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:32.730300 2872 hybrid_clock.cc:648] HybridClock initialized: now 1790713772730224 us; error 76 us; skew 500 ppm
I20260929 20:29:32.730635 2872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:32 dist-test-slave-p29l krb5kdc[2766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713772, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:32.734063 2872 init.cc:377] Logged in from keytab as kudu/127.28.244.129@KRBTEST.COM (short username kudu)
I20260929 20:29:32.735922 2872 webserver.cc:460] Webserver started at http://127.28.244.129:34019/ using document root <none> and password file <none>
I20260929 20:29:32.736703 2872 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:32.736861 2872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:32.737191 2872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:32.740324 2872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-0/data/instance:
uuid: "f298484818484dfb876ed5c2717afddf"
format_stamp: "Formatted at 2026-09-29 20:29:32 on dist-test-slave-p29l"
server_key: "054ce4764ed485be6484341dc698ae19"
server_key_iv: "2718e4f8352011e095263605e7ee288c"
server_key_version: "encryptionkey@0"
I20260929 20:29:32.741142 2872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance:
uuid: "f298484818484dfb876ed5c2717afddf"
format_stamp: "Formatted at 2026-09-29 20:29:32 on dist-test-slave-p29l"
server_key: "054ce4764ed485be6484341dc698ae19"
server_key_iv: "2718e4f8352011e095263605e7ee288c"
server_key_version: "encryptionkey@0"
I20260929 20:29:32.746599 2872 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:29:32.750514 2887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:32.752012 2872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:29:32.752200 2872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-0/wal
uuid: "f298484818484dfb876ed5c2717afddf"
format_stamp: "Formatted at 2026-09-29 20:29:32 on dist-test-slave-p29l"
server_key: "054ce4764ed485be6484341dc698ae19"
server_key_iv: "2718e4f8352011e095263605e7ee288c"
server_key_version: "encryptionkey@0"
I20260929 20:29:32.752525 2872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:32.766610 2872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:32.770632 2872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:32.770937 2872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:32.771992 2872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:29:32.773393 2872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:29:32.773473 2872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:32.773628 2872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:29:32.773695 2872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:32.807490 2872 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.129:33559
I20260929 20:29:32.807708 2999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.129:33559 every 8 connection(s)
I20260929 20:29:32.808911 2872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
I20260929 20:29:32.813465 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 2872
I20260929 20:29:32.813681 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance
I20260929 20:29:32.814325 29650 external_mini_cluster.cc:1428] Setting key 2f66ce5c64feaf944eae1e37ecb28433
Sep 29 20:29:32 dist-test-slave-p29l krb5kdc[2766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713772, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:29:32.832341 3002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:32.811568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:53679 (local address 127.28.244.190:41093)
0929 20:29:32.812058 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:32.812063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:32.813090 (+ 1027us) server_negotiation.cc:407] Connection header received
0929 20:29:32.815136 (+ 2046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:32.815142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:32.815260 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:32.815442 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:32.817530 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:32.818764 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:32.820504 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:32.821219 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:32.821378 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:32.825621 (+ 4243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:32.825626 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:32.825630 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:32.825664 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:32.828476 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:32.829466 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:32.829471 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:32.829473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:32.829595 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:32.830149 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:32.830153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:32.830155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:32.830581 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:32.830714 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:32.831738 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0929 20:29:32.832017 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":263,"threads_started":1}
I20260929 20:29:32.833938 3003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:32.812103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:41093 (local address 127.28.244.129:53679)
0929 20:29:32.812896 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:32.812933 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:32.814810 (+ 1877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:32.815794 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:32.815808 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:32.816675 (+ 867us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:32.817318 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:32.817342 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:32.818950 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:32.818955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:32.820350 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:32.820365 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:32.821453 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:32.821458 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:32.821666 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:32.822402 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:32.822426 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:32.825351 (+ 2925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:32.828633 (+ 3282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:32.828645 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:32.828668 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:32.829303 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:32.829714 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:32.829721 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:32.829727 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:32.829999 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:32.830735 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:32.830746 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:32.831271 (+ 525us) client_negotiation.cc:769] Sending connection context
0929 20:29:32.831659 (+ 388us) client_negotiation.cc:240] Negotiation successful
0929 20:29:32.832339 (+ 680us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":368,"threads_started":1}
I20260929 20:29:32.835568 3000 heartbeater.cc:347] Connected to a master server at 127.28.244.190:41093
I20260929 20:29:32.835965 3000 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.28.244.130@KRBTEST.COM; defaulting to no policy
I20260929 20:29:32.836856 3000 heartbeater.cc:511] Master 127.28.244.190:41093 requested a full tablet report, sending...
I20260929 20:29:32.839016 2811 ts_manager.cc:194] Registered new tserver with Master: f298484818484dfb876ed5c2717afddf (127.28.244.129:33559)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.130@KRBTEST.COM" created.
I20260929 20:29:32.840583 2811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.129@KRBTEST.COM'} at 127.28.244.129:53679
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab.
Entry for principal kudu/127.28.244.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab.
I20260929 20:29:32.864615 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.130:0
--local_ip_for_outbound_sockets=127.28.244.130
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:41093
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:38615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:29:33.219291 3006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:33.219602 3006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:33.219770 3006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:33.227629 3006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:33.227771 3006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:33.227962 3006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.130
I20260929 20:29:33.238606 3006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:38615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:41093
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.130
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:33.240079 3006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:33.241473 3006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:33.256157 3012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:33.256268 3011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:33.256264 3014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:33.257226 3006 server_base.cc:1034] running on GCE node
I20260929 20:29:33.257887 3006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:33.258912 3006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:33.260111 3006 hybrid_clock.cc:648] HybridClock initialized: now 1790713773260045 us; error 71 us; skew 500 ppm
I20260929 20:29:33.260434 3006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:33 dist-test-slave-p29l krb5kdc[2766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713773, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:33.263625 3006 init.cc:377] Logged in from keytab as kudu/127.28.244.130@KRBTEST.COM (short username kudu)
I20260929 20:29:33.265249 3006 webserver.cc:460] Webserver started at http://127.28.244.130:45977/ using document root <none> and password file <none>
I20260929 20:29:33.265959 3006 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:33.266106 3006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:33.266422 3006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:33.269480 3006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-1/data/instance:
uuid: "908e841860a347c888823035166bdc84"
format_stamp: "Formatted at 2026-09-29 20:29:33 on dist-test-slave-p29l"
server_key: "45f344b18dfca455ea0394db3c6213ae"
server_key_iv: "4455fdd1780cf50a73eb11ef3a5cb4b7"
server_key_version: "encryptionkey@0"
I20260929 20:29:33.270298 3006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance:
uuid: "908e841860a347c888823035166bdc84"
format_stamp: "Formatted at 2026-09-29 20:29:33 on dist-test-slave-p29l"
server_key: "45f344b18dfca455ea0394db3c6213ae"
server_key_iv: "4455fdd1780cf50a73eb11ef3a5cb4b7"
server_key_version: "encryptionkey@0"
I20260929 20:29:33.275779 3006 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 20:29:33.279452 3021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:33.280898 3006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 20:29:33.281087 3006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-1/wal
uuid: "908e841860a347c888823035166bdc84"
format_stamp: "Formatted at 2026-09-29 20:29:33 on dist-test-slave-p29l"
server_key: "45f344b18dfca455ea0394db3c6213ae"
server_key_iv: "4455fdd1780cf50a73eb11ef3a5cb4b7"
server_key_version: "encryptionkey@0"
I20260929 20:29:33.281298 3006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:33.304168 3006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:33.308491 3006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:33.308791 3006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:33.309801 3006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:29:33.311305 3006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:29:33.311410 3006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:33.311561 3006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:29:33.311632 3006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:33.346483 3006 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.130:37337
I20260929 20:29:33.346586 3133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.130:37337 every 8 connection(s)
I20260929 20:29:33.347895 3006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
I20260929 20:29:33.356513 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 3006
I20260929 20:29:33.356699 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance
I20260929 20:29:33.357280 29650 external_mini_cluster.cc:1428] Setting key 6fd96e9ba7d68e7fc029bef116483984
Sep 29 20:29:33 dist-test-slave-p29l krb5kdc[2766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713773, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:29:33.371451 3136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:33.350700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:50071 (local address 127.28.244.190:41093)
0929 20:29:33.351230 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:33.351236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:33.352374 (+ 1138us) server_negotiation.cc:407] Connection header received
0929 20:29:33.354272 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:33.354277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:33.354345 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:33.354446 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:33.356141 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:33.357293 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:33.359346 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:33.359888 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:33.360003 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:33.364491 (+ 4488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:33.364500 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:33.364508 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:33.364554 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:33.367758 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:33.368881 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:33.368885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:33.368888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:33.369045 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:33.369583 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:33.369586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:33.369588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:33.369861 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:33.370008 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:33.370989 (+ 981us) server_negotiation.cc:299] Negotiation successful
0929 20:29:33.371185 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":301,"threads_started":1}
I20260929 20:29:33.373246 3137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:33.351354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:41093 (local address 127.28.244.130:50071)
0929 20:29:33.352192 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:33.352231 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:33.354005 (+ 1774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:33.354674 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:33.354683 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:33.355258 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:33.355952 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:33.355974 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:33.357543 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:33.357547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:33.359176 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:33.359194 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:33.360134 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:33.360142 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:33.360491 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:33.361331 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:33.361368 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:33.364224 (+ 2856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:33.367984 (+ 3760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:33.367997 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:33.368020 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:33.368735 (+ 715us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:33.369169 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:33.369177 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:33.369184 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:33.369452 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:33.370029 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:33.370040 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:33.370559 (+ 519us) client_negotiation.cc:769] Sending connection context
0929 20:29:33.370869 (+ 310us) client_negotiation.cc:240] Negotiation successful
0929 20:29:33.371503 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":295,"threads_started":1}
I20260929 20:29:33.374831 3134 heartbeater.cc:347] Connected to a master server at 127.28.244.190:41093
I20260929 20:29:33.375186 3134 heartbeater.cc:464] Registering TS with master...
I20260929 20:29:33.376021 3134 heartbeater.cc:511] Master 127.28.244.190:41093 requested a full tablet report, sending...
I20260929 20:29:33.377750 2811 ts_manager.cc:194] Registered new tserver with Master: 908e841860a347c888823035166bdc84 (127.28.244.130:37337)
I20260929 20:29:33.378618 2811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.130@KRBTEST.COM'} at 127.28.244.130:50071
WARNING: no policy specified for kudu/127.28.244.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab.
Entry for principal kudu/127.28.244.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab.
I20260929 20:29:33.411831 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.131:0
--local_ip_for_outbound_sockets=127.28.244.131
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:41093
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:38615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:29:33.759701 3140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:33.760106 3140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:33.760275 3140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:33.767941 3140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:33.768096 3140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:33.768282 3140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.131
I20260929 20:29:33.778407 3140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:38615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:41093
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.131
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:33.779898 3140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:33.781322 3140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:33.795840 3146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:33.795917 3145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:33.796434 3148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:33.796725 3140 server_base.cc:1034] running on GCE node
I20260929 20:29:33.797396 3140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:33.798403 3140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:33.799647 3140 hybrid_clock.cc:648] HybridClock initialized: now 1790713773799598 us; error 52 us; skew 500 ppm
I20260929 20:29:33.800040 3140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:33 dist-test-slave-p29l krb5kdc[2766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713773, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:33.803248 3140 init.cc:377] Logged in from keytab as kudu/127.28.244.131@KRBTEST.COM (short username kudu)
I20260929 20:29:33.804967 3140 webserver.cc:460] Webserver started at http://127.28.244.131:41489/ using document root <none> and password file <none>
I20260929 20:29:33.805704 3140 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:33.805860 3140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:33.806183 3140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:33.809391 3140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-2/data/instance:
uuid: "75f336049ad44742a8a916e1cc9c8e77"
format_stamp: "Formatted at 2026-09-29 20:29:33 on dist-test-slave-p29l"
server_key: "5c289bed6a41659cccb01d7c77181d3f"
server_key_iv: "9e4fc273bc46411bda82ab51f43ad444"
server_key_version: "encryptionkey@0"
I20260929 20:29:33.810222 3140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance:
uuid: "75f336049ad44742a8a916e1cc9c8e77"
format_stamp: "Formatted at 2026-09-29 20:29:33 on dist-test-slave-p29l"
server_key: "5c289bed6a41659cccb01d7c77181d3f"
server_key_iv: "9e4fc273bc46411bda82ab51f43ad444"
server_key_version: "encryptionkey@0"
I20260929 20:29:33.816121 3140 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 20:29:33.820019 3155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:33.821647 3140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:29:33.821877 3140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-2/wal
uuid: "75f336049ad44742a8a916e1cc9c8e77"
format_stamp: "Formatted at 2026-09-29 20:29:33 on dist-test-slave-p29l"
server_key: "5c289bed6a41659cccb01d7c77181d3f"
server_key_iv: "9e4fc273bc46411bda82ab51f43ad444"
server_key_version: "encryptionkey@0"
I20260929 20:29:33.822131 3140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:33.843533 3000 heartbeater.cc:503] Master 127.28.244.190:41093 was elected leader, sending a full tablet report...
I20260929 20:29:33.844779 3140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:33.849740 3140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:33.850066 3140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:33.851153 3140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:29:33.852734 3140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:29:33.852847 3140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:33.853011 3140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:29:33.853084 3140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:33.888608 3140 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.131:40417
I20260929 20:29:33.888701 3267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.131:40417 every 8 connection(s)
I20260929 20:29:33.890033 3140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
I20260929 20:29:33.894537 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 3140
I20260929 20:29:33.894726 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance
I20260929 20:29:33.895299 29650 external_mini_cluster.cc:1428] Setting key 7602b1c7406b4fb6e69a37565d323715
Sep 29 20:29:33 dist-test-slave-p29l krb5kdc[2766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713773, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:29:33.911739 3270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:33.892961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:42291 (local address 127.28.244.190:41093)
0929 20:29:33.893447 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:33.893451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:33.894478 (+ 1027us) server_negotiation.cc:407] Connection header received
0929 20:29:33.896690 (+ 2212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:33.896696 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:33.896805 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:33.896923 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:33.898508 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:33.899281 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:33.900939 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:33.901347 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:33.901495 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:33.905686 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:33.905692 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:33.905695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:33.905731 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:33.908432 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:33.909246 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:33.909251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:33.909253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:33.909380 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:33.909868 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:33.909873 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:33.909875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:33.910169 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:33.910343 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:33.911108 (+ 765us) server_negotiation.cc:299] Negotiation successful
0929 20:29:33.911335 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":245,"threads_started":1}
I20260929 20:29:33.912849 3271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:33.893551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:41093 (local address 127.28.244.131:42291)
0929 20:29:33.894313 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:33.894350 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:33.896412 (+ 2062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:33.897187 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:33.897196 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:33.897731 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:33.898326 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:33.898344 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:33.899452 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:33.899457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:33.900789 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:33.900801 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:33.901583 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:33.901590 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:33.901843 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:33.902609 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:33.902638 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:33.905426 (+ 2788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:33.908600 (+ 3174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:33.908610 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:33.908629 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:33.909098 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:33.909511 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:33.909516 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:33.909520 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:33.909733 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:33.910311 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:33.910320 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:33.910714 (+ 394us) client_negotiation.cc:769] Sending connection context
0929 20:29:33.910994 (+ 280us) client_negotiation.cc:240] Negotiation successful
0929 20:29:33.911598 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":297,"threads_started":1}
I20260929 20:29:33.914348 3268 heartbeater.cc:347] Connected to a master server at 127.28.244.190:41093
I20260929 20:29:33.914714 3268 heartbeater.cc:464] Registering TS with master...
I20260929 20:29:33.915509 3268 heartbeater.cc:511] Master 127.28.244.190:41093 requested a full tablet report, sending...
I20260929 20:29:33.917064 2811 ts_manager.cc:194] Registered new tserver with Master: 75f336049ad44742a8a916e1cc9c8e77 (127.28.244.131:40417)
I20260929 20:29:33.917943 2811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.131@KRBTEST.COM'} at 127.28.244.131:42291
I20260929 20:29:33.923967 29650 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/krb5kdc/test-admin.keytab.
Sep 29 20:29:33 dist-test-slave-p29l krb5kdc[2766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:33.951289 29650 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 20:29:33.956940 3276 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:29:33.954155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37238 (local address 127.28.244.129:33559)
0929 20:29:33.954754 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:33.954761 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:33.954789 (+ 28us) server_negotiation.cc:407] Connection header received
0929 20:29:33.954853 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:33.954867 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:33.955162 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:33.955302 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:33.956434 (+ 1132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37238: BlockingRecv error: recv got EOF from 127.0.0.1:37238 (error 108)
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":299,"threads_started":1}
W20260929 20:29:33.956986 29650 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/krb5kdc/krb5cc' of type FILE
Sep 29 20:29:33 dist-test-slave-p29l krb5kdc[2766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:33.958702 29650 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 20:29:33 dist-test-slave-p29l krb5kdc[2766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:33.960175 29650 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 20:29:33 dist-test-slave-p29l krb5kdc[2766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.129@KRBTEST.COM
I20260929 20:29:33.977310 3276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:33.962085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37252 (local address 127.28.244.129:33559)
0929 20:29:33.962323 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:33.962330 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:33.962568 (+ 238us) server_negotiation.cc:407] Connection header received
0929 20:29:33.962803 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:33.962807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:33.962896 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:33.963010 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:33.964116 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:33.965102 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:33.966360 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:33.967026 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:33.967184 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:33.971208 (+ 4024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:33.971221 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:33.971237 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:33.971276 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:33.974217 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:33.975011 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:33.975019 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:33.975021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:33.975143 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:33.975551 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:33.975557 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:33.975560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:33.976019 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:33.976336 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:33.976681 (+ 345us) server_negotiation.cc:299] Negotiation successful
0929 20:29:33.977048 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260929 20:29:33.983493 3276 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:29:33.981622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37256 (local address 127.28.244.129:33559)
0929 20:29:33.981783 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:33.981788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:33.982174 (+ 386us) server_negotiation.cc:407] Connection header received
0929 20:29:33.982313 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:33.982317 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:33.982399 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:33.982484 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:33.983209 (+ 725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37256: BlockingRecv error: recv got EOF from 127.0.0.1:37256 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260929 20:29:33.983711 29650 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/krb5kdc/krb5cc' of type FILE
Sep 29 20:29:33 dist-test-slave-p29l krb5kdc[2766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:33.985416 29650 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 20:29:33 dist-test-slave-p29l krb5kdc[2766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:33.986937 29650 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 20:29:33 dist-test-slave-p29l krb5kdc[2766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.129@KRBTEST.COM
I20260929 20:29:34.001781 3276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:33.988373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37258 (local address 127.28.244.129:33559)
0929 20:29:33.988606 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:33.988613 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:33.988931 (+ 318us) server_negotiation.cc:407] Connection header received
0929 20:29:33.989144 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:33.989148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:33.989209 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:33.989371 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:33.990284 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:33.990987 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:33.992279 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:33.992752 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:33.992893 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:33.996698 (+ 3805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:33.996703 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:33.996706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:33.996753 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:33.999300 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:34.000030 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:34.000037 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:34.000040 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:34.000177 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:34.000621 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:34.000625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:34.000627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:34.000899 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:34.001093 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:34.001353 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 20:29:34.001557 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260929 20:29:34.006181 3276 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:29:34.004575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37272 (local address 127.28.244.129:33559)
0929 20:29:34.004777 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:34.004782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:34.004908 (+ 126us) server_negotiation.cc:407] Connection header received
0929 20:29:34.005050 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:34.005053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:34.005119 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:34.005199 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:34.005965 (+ 766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37272: BlockingRecv error: recv got EOF from 127.0.0.1:37272 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260929 20:29:34.006299 29650 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713721201454-29650-0/krb5kdc/krb5cc' of type FILE
Sep 29 20:29:34 dist-test-slave-p29l krb5kdc[2766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:34.008008 29650 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 20:29:34 dist-test-slave-p29l krb5kdc[2766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:34.009501 29650 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 20:29:34 dist-test-slave-p29l krb5kdc[2766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.129@KRBTEST.COM
I20260929 20:29:34.024780 3276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:34.010951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37288 (local address 127.28.244.129:33559)
0929 20:29:34.011181 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:34.011187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:34.011464 (+ 277us) server_negotiation.cc:407] Connection header received
0929 20:29:34.011660 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:34.011668 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:34.011732 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:34.011862 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:34.012786 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:34.013515 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:34.014764 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:34.015195 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:34.015341 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:34.019172 (+ 3831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:34.019177 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:34.019181 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:34.019217 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:34.022074 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:34.022860 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:34.022866 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:34.022870 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:34.023030 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:34.023445 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:34.023449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:34.023451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:34.023744 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:34.024083 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:34.024286 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 20:29:34.024503 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 20:29:34.026340 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 2872
I20260929 20:29:34.039288 3276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:34.027177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37304 (local address 127.28.244.129:33559)
0929 20:29:34.027348 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:34.027356 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:34.027940 (+ 584us) server_negotiation.cc:407] Connection header received
0929 20:29:34.028053 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:34.028056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:34.028114 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:34.028200 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:34.029163 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:34.029869 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:34.031188 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:34.031672 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:34.031785 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:34.034087 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:34.034091 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:34.034095 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:34.034127 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:34.036820 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:34.037471 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:34.037476 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:34.037478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:34.037627 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:34.037983 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:34.037987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:34.037989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:34.038239 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:34.038482 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:34.038845 (+ 363us) server_negotiation.cc:299] Negotiation successful
0929 20:29:34.039023 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 20:29:34.040093 2995 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:29:34.125321 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 2872
I20260929 20:29:34.142761 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 3006
Sep 29 20:29:34 dist-test-slave-p29l krb5kdc[2766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.130@KRBTEST.COM
I20260929 20:29:34.162773 3289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:34.143711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53134 (local address 127.28.244.130:37337)
0929 20:29:34.144282 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:34.144289 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:34.144314 (+ 25us) server_negotiation.cc:407] Connection header received
0929 20:29:34.144378 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:34.144387 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:34.144608 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:34.144745 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:34.146178 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:34.148492 (+ 2314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:34.150488 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:34.151634 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:34.151783 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:34.156829 (+ 5046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:34.156849 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:34.156865 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:34.156922 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:34.159721 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:34.160551 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:34.160557 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:34.160560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:34.160693 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:34.161098 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:34.161101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:34.161103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:34.161511 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:34.161737 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:34.162117 (+ 380us) server_negotiation.cc:299] Negotiation successful
0929 20:29:34.162326 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":305,"threads_started":1}
I20260929 20:29:34.164180 3129 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:29:34.254135 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 3006
I20260929 20:29:34.271270 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 3140
Sep 29 20:29:34 dist-test-slave-p29l krb5kdc[2766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.131@KRBTEST.COM
I20260929 20:29:34.288872 3291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:34.272681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54392 (local address 127.28.244.131:40417)
0929 20:29:34.273460 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:34.273468 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:34.273497 (+ 29us) server_negotiation.cc:407] Connection header received
0929 20:29:34.273560 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:34.273567 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:34.273785 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:34.273942 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:34.275064 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:34.276250 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:34.277551 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:34.278125 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:34.278224 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:34.282435 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:34.282448 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:34.282464 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:34.282509 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:34.286106 (+ 3597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:34.286845 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:34.286853 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:34.286855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:34.286997 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:34.287323 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:34.287326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:34.287328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:34.287735 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:34.288072 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:34.288306 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 20:29:34.288579 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":420,"threads_started":1}
I20260929 20:29:34.290268 3263 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:29:34.374121 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 3140
I20260929 20:29:34.390940 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 2779
I20260929 20:29:34.392284 2841 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:29:34.497867 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 2779
2026-09-29T20:29:34Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4092 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:29:34 dist-test-slave-p29l krb5kdc[3296](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:29:34 dist-test-slave-p29l krb5kdc[3296](info): set up 2 sockets
Sep 29 20:29:34 dist-test-slave-p29l krb5kdc[3296](info): commencing operation
krb5kdc: starting...
W20260929 20:29:35.600194 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:29:35 dist-test-slave-p29l krb5kdc[3296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:29:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:29:35Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.244.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.244.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.244.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.190.keytab.
Entry for principal oryx/127.28.244.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.190.keytab.
I20260929 20:29:35.827462 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:39805
--webserver_interface=127.28.244.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.190.keytab
--principal=oryx/127.28.244.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:39805
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:29:36.162351 3309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:36.162739 3309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:36.162894 3309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:36.170557 3309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:29:36.170688 3309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:36.170776 3309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:36.170866 3309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:29:36.170948 3309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:29:36.181478 3309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:39805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.190.keytab
--principal=oryx/127.28.244.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:39805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:36.182928 3309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:36.184381 3309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:36.197075 3314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:36.197188 3315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:36.197052 3317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:36.197926 3309 server_base.cc:1034] running on GCE node
I20260929 20:29:36.198809 3309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:36.200464 3309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:36.201696 3309 hybrid_clock.cc:648] HybridClock initialized: now 1790713776201646 us; error 33 us; skew 500 ppm
I20260929 20:29:36.202219 3309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:36 dist-test-slave-p29l krb5kdc[3296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713776, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.244.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:36.209250 3309 init.cc:377] Logged in from keytab as oryx/127.28.244.190@KRBTEST.COM (short username oryx)
I20260929 20:29:36.211227 3309 webserver.cc:460] Webserver started at http://127.28.244.190:35383/ using document root <none> and password file <none>
I20260929 20:29:36.212150 3309 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:36.212328 3309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:36.212667 3309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:36.216171 3309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "c23310ae5174458f8743b50aefe320bc"
format_stamp: "Formatted at 2026-09-29 20:29:36 on dist-test-slave-p29l"
server_key: "3c8da94c8778f5d7ea1cff1811ec22f1"
server_key_iv: "8d431cde30f359f21aaeeba9d190f35b"
server_key_version: "encryptionkey@0"
I20260929 20:29:36.217069 3309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "c23310ae5174458f8743b50aefe320bc"
format_stamp: "Formatted at 2026-09-29 20:29:36 on dist-test-slave-p29l"
server_key: "3c8da94c8778f5d7ea1cff1811ec22f1"
server_key_iv: "8d431cde30f359f21aaeeba9d190f35b"
server_key_version: "encryptionkey@0"
I20260929 20:29:36.223023 3309 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 20:29:36.227258 3324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:36.229166 3309 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 20:29:36.229414 3309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "c23310ae5174458f8743b50aefe320bc"
format_stamp: "Formatted at 2026-09-29 20:29:36 on dist-test-slave-p29l"
server_key: "3c8da94c8778f5d7ea1cff1811ec22f1"
server_key_iv: "8d431cde30f359f21aaeeba9d190f35b"
server_key_version: "encryptionkey@0"
I20260929 20:29:36.229687 3309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:36.254288 3309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:36.258937 3309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:36.259258 3309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:36.278998 3309 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:39805
I20260929 20:29:36.279047 3375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:39805 every 8 connection(s)
I20260929 20:29:36.280471 3309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:29:36.284369 3376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:36.289163 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 3309
I20260929 20:29:36.289379 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:29:36.290225 29650 external_mini_cluster.cc:1428] Setting key 16a78366ad52dffdc036d5323bc608db
I20260929 20:29:36.296643 3376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc: Bootstrap starting.
I20260929 20:29:36.300351 3376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:36.301518 3376 log.cc:824] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc: Log is configured to *not* fsync() on all Append() calls
I20260929 20:29:36.305058 3376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc: No bootstrap required, opened a new log
Sep 29 20:29:36 dist-test-slave-p29l krb5kdc[3296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM
I20260929 20:29:36.312994 3376 raft_consensus.cc:348] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } }
I20260929 20:29:36.313405 3376 raft_consensus.cc:374] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:36.313540 3376 raft_consensus.cc:729] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c23310ae5174458f8743b50aefe320bc, State: Initialized, Role: FOLLOWER
I20260929 20:29:36.313973 3379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:36.292396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44510 (local address 127.28.244.190:39805)
0929 20:29:36.293189 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:36.293201 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:36.293242 (+ 41us) server_negotiation.cc:407] Connection header received
0929 20:29:36.295000 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:36.295035 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:36.295419 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:36.295743 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:36.297632 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:36.298992 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:36.300832 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:36.301500 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:36.301684 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:36.306351 (+ 4667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:36.306359 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:36.306374 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:36.306409 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:36.309631 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:36.310347 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:36.310354 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:36.310361 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:36.310533 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:36.310950 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:36.310953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:36.310955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:36.311540 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:36.311905 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:36.312357 (+ 452us) server_negotiation.cc:299] Negotiation successful
0929 20:29:36.312790 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":371,"threads_started":1}
I20260929 20:29:36.314308 3376 consensus_queue.cc:260] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } }
I20260929 20:29:36.314587 3376 raft_consensus.cc:388] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:29:36.314743 3376 raft_consensus.cc:482] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:29:36.314939 3376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:36.317492 3376 raft_consensus.cc:504] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } }
I20260929 20:29:36.317996 3376 leader_election.cc:302] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c23310ae5174458f8743b50aefe320bc; no voters:
I20260929 20:29:36.318559 3376 leader_election.cc:288] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:29:36.318686 3381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:29:36.319386 3381 raft_consensus.cc:686] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [term 1 LEADER]: Becoming Leader. State: Replica: c23310ae5174458f8743b50aefe320bc, State: Running, Role: LEADER
I20260929 20:29:36.320029 3381 consensus_queue.cc:237] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } }
I20260929 20:29:36.320912 3376 sys_catalog.cc:564] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:29:36.323477 3383 sys_catalog.cc:455] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader c23310ae5174458f8743b50aefe320bc. Latest consensus state: current_term: 1 leader_uuid: "c23310ae5174458f8743b50aefe320bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } }
I20260929 20:29:36.323441 3382 sys_catalog.cc:455] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c23310ae5174458f8743b50aefe320bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } }
I20260929 20:29:36.323982 3382 sys_catalog.cc:458] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [sys.catalog]: This master's current role is: LEADER
I20260929 20:29:36.323982 3383 sys_catalog.cc:458] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [sys.catalog]: This master's current role is: LEADER
I20260929 20:29:36.327127 3389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:29:36.332527 3389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:29:36.342087 3389 catalog_manager.cc:1347] Generated new cluster ID: bb52180e1176467eb311d00f60c824a6
I20260929 20:29:36.342216 3389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:29:36.354904 3389 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:29:36.356843 3389 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:29:36.369846 3389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc: Generated new TSK 0
I20260929 20:29:36.370808 3389 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.244.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.244.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.244.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.129.keytab.
Entry for principal oryx/127.28.244.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.129.keytab.
I20260929 20:29:36.431391 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.129:0
--local_ip_for_outbound_sockets=127.28.244.129
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:39805
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.129.keytab
--principal=oryx/127.28.244.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:29:36.767707 3402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:36.768105 3402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:36.768278 3402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:36.775656 3402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:36.775813 3402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:36.776006 3402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.129
I20260929 20:29:36.786080 3402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.129.keytab
--principal=oryx/127.28.244.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:39805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.129
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:36.787428 3402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:36.788830 3402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:36.801972 3407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:36.801884 3408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:36.801928 3410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:36.802975 3402 server_base.cc:1034] running on GCE node
I20260929 20:29:36.803628 3402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:36.804633 3402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:36.805878 3402 hybrid_clock.cc:648] HybridClock initialized: now 1790713776805836 us; error 52 us; skew 500 ppm
I20260929 20:29:36.806222 3402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:36 dist-test-slave-p29l krb5kdc[3296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713776, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.244.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:36.809896 3402 init.cc:377] Logged in from keytab as oryx/127.28.244.129@KRBTEST.COM (short username oryx)
I20260929 20:29:36.811357 3402 webserver.cc:460] Webserver started at http://127.28.244.129:35433/ using document root <none> and password file <none>
I20260929 20:29:36.812163 3402 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:36.812299 3402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:36.812637 3402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:36.815568 3402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-0/data/instance:
uuid: "e9b791bada31439da862ec8bf5a97488"
format_stamp: "Formatted at 2026-09-29 20:29:36 on dist-test-slave-p29l"
server_key: "56eb302287518ab0b0042db4326f6e84"
server_key_iv: "f3b3802acfbe466ac1c963ccae87482f"
server_key_version: "encryptionkey@0"
I20260929 20:29:36.816496 3402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance:
uuid: "e9b791bada31439da862ec8bf5a97488"
format_stamp: "Formatted at 2026-09-29 20:29:36 on dist-test-slave-p29l"
server_key: "56eb302287518ab0b0042db4326f6e84"
server_key_iv: "f3b3802acfbe466ac1c963ccae87482f"
server_key_version: "encryptionkey@0"
I20260929 20:29:36.821729 3402 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:29:36.825275 3417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:36.826706 3402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:29:36.826891 3402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-0/wal
uuid: "e9b791bada31439da862ec8bf5a97488"
format_stamp: "Formatted at 2026-09-29 20:29:36 on dist-test-slave-p29l"
server_key: "56eb302287518ab0b0042db4326f6e84"
server_key_iv: "f3b3802acfbe466ac1c963ccae87482f"
server_key_version: "encryptionkey@0"
I20260929 20:29:36.827100 3402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:36.856535 3402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:36.861366 3402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:36.861685 3402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:36.862704 3402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:29:36.864229 3402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:29:36.864305 3402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:36.864444 3402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:29:36.864523 3402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:36.898120 3402 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.129:38387
I20260929 20:29:36.898162 3529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.129:38387 every 8 connection(s)
I20260929 20:29:36.899466 3402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
I20260929 20:29:36.905306 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 3402
I20260929 20:29:36.905526 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance
I20260929 20:29:36.906196 29650 external_mini_cluster.cc:1428] Setting key 7cc11a08ad7ba09a9a2e079e184544ae
Sep 29 20:29:36 dist-test-slave-p29l krb5kdc[3296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713776, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.244.129@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM
I20260929 20:29:36.920523 3532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:36.902054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:46161 (local address 127.28.244.190:39805)
0929 20:29:36.902468 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:36.902473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:36.903449 (+ 976us) server_negotiation.cc:407] Connection header received
0929 20:29:36.905413 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:36.905420 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:36.905536 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:36.905659 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:36.907467 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:36.908326 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:36.909932 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:36.910671 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:36.910816 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:36.914510 (+ 3694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:36.914515 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:36.914518 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:36.914550 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:36.917207 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:36.918144 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:36.918147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:36.918150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:36.918264 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:36.918755 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:36.918758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:36.918760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:36.919019 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:36.919127 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:36.920077 (+ 950us) server_negotiation.cc:299] Negotiation successful
0929 20:29:36.920257 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":204,"threads_started":1}
I20260929 20:29:36.922232 3533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:36.902624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.28.244.129:46161)
0929 20:29:36.903291 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:36.903334 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:36.905093 (+ 1759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:36.905912 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:36.905921 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:36.906500 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:36.907264 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:36.907292 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:36.908496 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:36.908500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:36.909767 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:36.909780 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:36.910881 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:36.910886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:36.911092 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:36.911800 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:36.911866 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:36.914277 (+ 2411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:36.917364 (+ 3087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:36.917376 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:36.917397 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:36.918028 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:36.918370 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:36.918378 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:36.918384 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:36.918641 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:36.919147 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:36.919158 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:36.919640 (+ 482us) client_negotiation.cc:769] Sending connection context
0929 20:29:36.919988 (+ 348us) client_negotiation.cc:240] Negotiation successful
0929 20:29:36.920631 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":270,"threads_started":1}
I20260929 20:29:36.923907 3530 heartbeater.cc:347] Connected to a master server at 127.28.244.190:39805
I20260929 20:29:36.924279 3530 heartbeater.cc:464] Registering TS with master...
I20260929 20:29:36.925071 3530 heartbeater.cc:511] Master 127.28.244.190:39805 requested a full tablet report, sending...
I20260929 20:29:36.927462 3341 ts_manager.cc:194] Registered new tserver with Master: e9b791bada31439da862ec8bf5a97488 (127.28.244.129:38387)
WARNING: no policy specified for oryx/127.28.244.130@KRBTEST.COM; defaulting to no policy
I20260929 20:29:36.929104 3341 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.244.129@KRBTEST.COM'} at 127.28.244.129:46161
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.244.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.244.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.130.keytab.
Entry for principal oryx/127.28.244.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.130.keytab.
I20260929 20:29:36.955039 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.130:0
--local_ip_for_outbound_sockets=127.28.244.130
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:39805
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.130.keytab
--principal=oryx/127.28.244.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:29:37.284564 3536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:37.284907 3536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:37.285084 3536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:37.292493 3536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:37.292655 3536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:37.292835 3536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.130
I20260929 20:29:37.302824 3536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.130.keytab
--principal=oryx/127.28.244.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:39805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.130
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:37.304344 3536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:37.305673 3536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:37.319094 3541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:37.319020 3542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:37.319054 3544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:37.319693 3536 server_base.cc:1034] running on GCE node
I20260929 20:29:37.320421 3536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:37.321399 3536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:37.322628 3536 hybrid_clock.cc:648] HybridClock initialized: now 1790713777322574 us; error 35 us; skew 500 ppm
I20260929 20:29:37.322997 3536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:37 dist-test-slave-p29l krb5kdc[3296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713777, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.244.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:37.326323 3536 init.cc:377] Logged in from keytab as oryx/127.28.244.130@KRBTEST.COM (short username oryx)
I20260929 20:29:37.327962 3536 webserver.cc:460] Webserver started at http://127.28.244.130:38871/ using document root <none> and password file <none>
I20260929 20:29:37.328681 3536 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:37.328812 3536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:37.329116 3536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:37.332284 3536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-1/data/instance:
uuid: "5c245511f25e4a97901c08fdf79dfe74"
format_stamp: "Formatted at 2026-09-29 20:29:37 on dist-test-slave-p29l"
server_key: "07a04d4127834047663414dc06ab78a8"
server_key_iv: "664ab6b21b5f46defdb37c015fa58a8d"
server_key_version: "encryptionkey@0"
I20260929 20:29:37.333086 3536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance:
uuid: "5c245511f25e4a97901c08fdf79dfe74"
format_stamp: "Formatted at 2026-09-29 20:29:37 on dist-test-slave-p29l"
server_key: "07a04d4127834047663414dc06ab78a8"
server_key_iv: "664ab6b21b5f46defdb37c015fa58a8d"
server_key_version: "encryptionkey@0"
I20260929 20:29:37.338475 3536 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:29:37.341995 3551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:37.343397 3536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:29:37.343585 3536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-1/wal
uuid: "5c245511f25e4a97901c08fdf79dfe74"
format_stamp: "Formatted at 2026-09-29 20:29:37 on dist-test-slave-p29l"
server_key: "07a04d4127834047663414dc06ab78a8"
server_key_iv: "664ab6b21b5f46defdb37c015fa58a8d"
server_key_version: "encryptionkey@0"
I20260929 20:29:37.343802 3536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:37.355965 3536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:37.360476 3536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:37.360770 3536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:37.361708 3536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:29:37.363042 3536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:29:37.363153 3536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:37.363307 3536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:29:37.363382 3536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:37.397550 3536 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.130:40059
I20260929 20:29:37.397572 3663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.130:40059 every 8 connection(s)
I20260929 20:29:37.398869 3536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
I20260929 20:29:37.406028 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 3536
I20260929 20:29:37.406217 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance
I20260929 20:29:37.406675 29650 external_mini_cluster.cc:1428] Setting key 2d8a676b0da96a6d4c1e3ef62c815282
Sep 29 20:29:37 dist-test-slave-p29l krb5kdc[3296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713777, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.244.130@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM
I20260929 20:29:37.420449 3532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:37.401576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:53061 (local address 127.28.244.190:39805)
0929 20:29:37.401761 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:37.401766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:37.402926 (+ 1160us) server_negotiation.cc:407] Connection header received
0929 20:29:37.404867 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:37.404873 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:37.404981 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:37.405098 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:37.406782 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:37.407674 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:37.409757 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:37.410234 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:37.410380 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:37.414433 (+ 4053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:37.414438 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:37.414442 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:37.414478 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:37.417229 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:37.418025 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:37.418031 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:37.418033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:37.418159 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:37.418617 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:37.418622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:37.418624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:37.418900 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:37.419069 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:37.419852 (+ 783us) server_negotiation.cc:299] Negotiation successful
0929 20:29:37.420089 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 20:29:37.421624 3666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:37.402056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.28.244.130:53061)
0929 20:29:37.402752 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:37.402795 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:37.404575 (+ 1780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:37.405344 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:37.405352 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:37.405884 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:37.406594 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:37.406614 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:37.407874 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:37.407879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:37.409574 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:37.409588 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:37.410490 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:37.410498 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:37.410758 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:37.411487 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:37.411511 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:37.414165 (+ 2654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:37.417405 (+ 3240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:37.417414 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:37.417433 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:37.417878 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:37.418276 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:37.418281 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:37.418284 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:37.418486 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:37.419062 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:37.419071 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:37.419446 (+ 375us) client_negotiation.cc:769] Sending connection context
0929 20:29:37.419722 (+ 276us) client_negotiation.cc:240] Negotiation successful
0929 20:29:37.420330 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":272,"threads_started":1}
I20260929 20:29:37.423358 3664 heartbeater.cc:347] Connected to a master server at 127.28.244.190:39805
I20260929 20:29:37.423718 3664 heartbeater.cc:464] Registering TS with master...
I20260929 20:29:37.424494 3664 heartbeater.cc:511] Master 127.28.244.190:39805 requested a full tablet report, sending...
I20260929 20:29:37.426301 3341 ts_manager.cc:194] Registered new tserver with Master: 5c245511f25e4a97901c08fdf79dfe74 (127.28.244.130:40059)
WARNING: no policy specified for oryx/127.28.244.131@KRBTEST.COM; defaulting to no policy
I20260929 20:29:37.427500 3341 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.244.130@KRBTEST.COM'} at 127.28.244.130:53061
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.244.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.244.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.131.keytab.
Entry for principal oryx/127.28.244.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.131.keytab.
I20260929 20:29:37.456228 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.131:0
--local_ip_for_outbound_sockets=127.28.244.131
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:39805
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.131.keytab
--principal=oryx/127.28.244.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:29:37.797926 3669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:37.798204 3669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:37.798400 3669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:37.805775 3669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:37.805887 3669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:37.806087 3669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.131
I20260929 20:29:37.816032 3669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.131.keytab
--principal=oryx/127.28.244.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:39805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.131
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:37.817379 3669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:37.818722 3669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:37.832584 3674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:37.832623 3677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:37.832542 3675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:37.833592 3669 server_base.cc:1034] running on GCE node
I20260929 20:29:37.834256 3669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:37.835309 3669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:37.836563 3669 hybrid_clock.cc:648] HybridClock initialized: now 1790713777836498 us; error 50 us; skew 500 ppm
I20260929 20:29:37.836895 3669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:37 dist-test-slave-p29l krb5kdc[3296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713777, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.244.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:37.840140 3669 init.cc:377] Logged in from keytab as oryx/127.28.244.131@KRBTEST.COM (short username oryx)
I20260929 20:29:37.841820 3669 webserver.cc:460] Webserver started at http://127.28.244.131:45271/ using document root <none> and password file <none>
I20260929 20:29:37.842509 3669 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:37.842614 3669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:37.842870 3669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:37.845911 3669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-2/data/instance:
uuid: "7a7f8f2d242848f1af37b960ac84c810"
format_stamp: "Formatted at 2026-09-29 20:29:37 on dist-test-slave-p29l"
server_key: "3d9b19663d4490f50e852c7846fdbfdd"
server_key_iv: "8596d006e7470b62ab9b75e0403140bc"
server_key_version: "encryptionkey@0"
I20260929 20:29:37.846714 3669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance:
uuid: "7a7f8f2d242848f1af37b960ac84c810"
format_stamp: "Formatted at 2026-09-29 20:29:37 on dist-test-slave-p29l"
server_key: "3d9b19663d4490f50e852c7846fdbfdd"
server_key_iv: "8596d006e7470b62ab9b75e0403140bc"
server_key_version: "encryptionkey@0"
I20260929 20:29:37.852114 3669 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 20:29:37.855455 3684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:37.857010 3669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:29:37.857179 3669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-2/wal
uuid: "7a7f8f2d242848f1af37b960ac84c810"
format_stamp: "Formatted at 2026-09-29 20:29:37 on dist-test-slave-p29l"
server_key: "3d9b19663d4490f50e852c7846fdbfdd"
server_key_iv: "8596d006e7470b62ab9b75e0403140bc"
server_key_version: "encryptionkey@0"
I20260929 20:29:37.857368 3669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:37.874806 3669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:37.879427 3669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:37.879769 3669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:37.880796 3669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:29:37.882273 3669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:29:37.882349 3669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:37.882472 3669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:29:37.882535 3669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:37.914981 3669 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.131:36837
I20260929 20:29:37.915016 3796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.131:36837 every 8 connection(s)
I20260929 20:29:37.916369 3669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
I20260929 20:29:37.918083 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 3669
I20260929 20:29:37.918258 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance
I20260929 20:29:37.918872 29650 external_mini_cluster.cc:1428] Setting key 17b1334c176ebadf24af06526cd795f7
Sep 29 20:29:37 dist-test-slave-p29l krb5kdc[3296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713777, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.244.131@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM
I20260929 20:29:37.932673 3530 heartbeater.cc:503] Master 127.28.244.190:39805 was elected leader, sending a full tablet report...
I20260929 20:29:37.938377 3532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:37.919179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:39199 (local address 127.28.244.190:39805)
0929 20:29:37.919351 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:37.919356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:37.920984 (+ 1628us) server_negotiation.cc:407] Connection header received
0929 20:29:37.923200 (+ 2216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:37.923206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:37.923293 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:37.923410 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:37.925044 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:37.925802 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:37.927487 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:37.928058 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:37.928198 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:37.931931 (+ 3733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:37.931936 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:37.931939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:37.931991 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:37.935537 (+ 3546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:37.936352 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:37.936356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:37.936358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:37.936483 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:37.936869 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:37.936872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:37.936874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:37.937153 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:37.937268 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:37.937949 (+ 681us) server_negotiation.cc:299] Negotiation successful
0929 20:29:37.938130 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 20:29:37.939446 3799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:37.919811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.28.244.131:39199)
0929 20:29:37.920805 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:37.920861 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:37.922928 (+ 2067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:37.923671 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:37.923679 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:37.924253 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:37.924873 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:37.924891 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:37.925956 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:37.925961 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:37.927321 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:37.927333 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:37.928298 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:37.928306 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:37.928566 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:37.929295 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:37.929319 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:37.931662 (+ 2343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:37.935685 (+ 4023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:37.935693 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:37.935708 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:37.936240 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:37.936576 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:37.936579 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:37.936582 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:37.936771 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:37.937265 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:37.937271 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:37.937624 (+ 353us) client_negotiation.cc:769] Sending connection context
0929 20:29:37.937849 (+ 225us) client_negotiation.cc:240] Negotiation successful
0929 20:29:37.938274 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":369,"threads_started":1}
I20260929 20:29:37.940938 3797 heartbeater.cc:347] Connected to a master server at 127.28.244.190:39805
I20260929 20:29:37.941303 3797 heartbeater.cc:464] Registering TS with master...
I20260929 20:29:37.942117 3797 heartbeater.cc:511] Master 127.28.244.190:39805 requested a full tablet report, sending...
I20260929 20:29:37.943661 3340 ts_manager.cc:194] Registered new tserver with Master: 7a7f8f2d242848f1af37b960ac84c810 (127.28.244.131:36837)
I20260929 20:29:37.944593 3340 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.244.131@KRBTEST.COM'} at 127.28.244.131:39199
I20260929 20:29:37.948000 29650 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.244.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.244.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.244.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.189.keytab.
Entry for principal oryx/127.28.244.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.189.keytab.
I20260929 20:29:37.995313 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.189:44795
--webserver_interface=127.28.244.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.189.keytab
--principal=oryx/127.28.244.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:39805,127.28.244.189:44795
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:29:38.334172 3802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:38.334537 3802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:38.334686 3802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:38.342219 3802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:29:38.342384 3802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:38.342499 3802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:38.342600 3802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:29:38.342679 3802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:29:38.353559 3802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:39805,127.28.244.189:44795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.189.keytab
--principal=oryx/127.28.244.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.189:44795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.244.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
W20260929 20:29:38.354003 3802 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.28.244.190:39805,127.28.244.189:44795'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 20:29:38.355108 3802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:38.356490 3802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:38.368834 3808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:38.368786 3807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:38.368942 3810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:38.370029 3802 server_base.cc:1034] running on GCE node
I20260929 20:29:38.370814 3802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:38.372345 3802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:38.373569 3802 hybrid_clock.cc:648] HybridClock initialized: now 1790713778373507 us; error 33 us; skew 500 ppm
I20260929 20:29:38.373924 3802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:38 dist-test-slave-p29l krb5kdc[3296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713778, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.244.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:38.377156 3802 init.cc:377] Logged in from keytab as oryx/127.28.244.189@KRBTEST.COM (short username oryx)
I20260929 20:29:38.378687 3802 webserver.cc:460] Webserver started at http://127.28.244.189:42347/ using document root <none> and password file <none>
I20260929 20:29:38.379382 3802 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:38.379472 3802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:38.379734 3802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:38.382783 3802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-1/data/instance:
uuid: "985844716800427a89b43c509229630c"
format_stamp: "Formatted at 2026-09-29 20:29:38 on dist-test-slave-p29l"
server_key: "ec1ce62dfbe74765c4267aa54241d54f"
server_key_iv: "a11981dd1b4258fa59fce9e1be3dbf06"
server_key_version: "encryptionkey@0"
I20260929 20:29:38.383509 3802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-1/wal/instance:
uuid: "985844716800427a89b43c509229630c"
format_stamp: "Formatted at 2026-09-29 20:29:38 on dist-test-slave-p29l"
server_key: "ec1ce62dfbe74765c4267aa54241d54f"
server_key_iv: "a11981dd1b4258fa59fce9e1be3dbf06"
server_key_version: "encryptionkey@0"
I20260929 20:29:38.388777 3802 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 20:29:38.392118 3817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:38.393523 3802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:29:38.393712 3802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-1/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-1/wal
uuid: "985844716800427a89b43c509229630c"
format_stamp: "Formatted at 2026-09-29 20:29:38 on dist-test-slave-p29l"
server_key: "ec1ce62dfbe74765c4267aa54241d54f"
server_key_iv: "a11981dd1b4258fa59fce9e1be3dbf06"
server_key_version: "encryptionkey@0"
I20260929 20:29:38.393947 3802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:38.414888 3802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:38.418936 3802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:38.419219 3802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:38.430634 3664 heartbeater.cc:503] Master 127.28.244.190:39805 was elected leader, sending a full tablet report...
I20260929 20:29:38.438674 3802 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.189:44795
I20260929 20:29:38.438735 3868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.189:44795 every 8 connection(s)
I20260929 20:29:38.440182 3802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-1/data/info.pb
I20260929 20:29:38.443976 3869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:38.446182 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 3802
I20260929 20:29:38.453763 3869 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:29:38 dist-test-slave-p29l krb5kdc[3296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.244.189@KRBTEST.COM
Sep 29 20:29:38 dist-test-slave-p29l krb5kdc[3296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713778, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.244.189@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM
I20260929 20:29:38.471082 3872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:38.448594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34234 (local address 127.28.244.189:44795)
0929 20:29:38.449483 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:38.449495 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:38.449535 (+ 40us) server_negotiation.cc:407] Connection header received
0929 20:29:38.451087 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:38.451110 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:38.451498 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:38.451870 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:38.453700 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:38.455105 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:38.456617 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:38.457737 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:38.457908 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:38.462466 (+ 4558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:38.462474 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:38.462484 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:38.462520 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:38.466111 (+ 3591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:38.466891 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:38.466900 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:38.466910 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:38.467117 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:38.467556 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:38.467561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:38.467565 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:38.468405 (+ 840us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:38.468736 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:38.469187 (+ 451us) server_negotiation.cc:299] Negotiation successful
0929 20:29:38.469659 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":431,"threads_started":1}
I20260929 20:29:38.473062 3873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:38.456416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33334 (local address 127.28.244.190:39805)
0929 20:29:38.457842 (+ 1426us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:38.457848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:38.457865 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:29:38.457922 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:38.457927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:38.458016 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:38.458132 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:38.459625 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:38.460450 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:38.462137 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:38.462626 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:38.462718 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:38.466553 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:38.466558 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:38.466562 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:38.466597 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:38.470102 (+ 3505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:38.470822 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:38.470828 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:38.470831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:38.470992 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:38.471419 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:38.471423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:38.471426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:38.471752 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:38.471978 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:38.472466 (+ 488us) server_negotiation.cc:299] Negotiation successful
0929 20:29:38.472697 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1284,"thread_start_us":1161,"threads_started":1}
I20260929 20:29:38.473158 3874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:38.456406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.0.0.1:33334)
0929 20:29:38.457214 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:38.457265 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:38.457406 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:38.458233 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:38.458242 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:38.459107 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:38.459457 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:38.459471 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:38.460585 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:38.460591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:38.461985 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:38.461998 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:38.462798 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:38.462806 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:38.463037 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:38.463570 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:38.463600 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:38.466343 (+ 2743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:38.470240 (+ 3897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:38.470253 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:38.470258 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:38.470696 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:38.471114 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:38.471117 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:38.471119 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:38.471282 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:38.471940 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:38.471948 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:38.472133 (+ 185us) client_negotiation.cc:769] Sending connection context
0929 20:29:38.472358 (+ 225us) client_negotiation.cc:240] Negotiation successful
0929 20:29:38.472690 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":365,"threads_started":1}
I20260929 20:29:38.474956 3869 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:29:38 dist-test-slave-p29l krb5kdc[3296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713778, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.244.189@KRBTEST.COM for oryx/127.28.244.189@KRBTEST.COM
I20260929 20:29:38.486927 3874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:38.475795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:44795 (local address 127.0.0.1:34242)
0929 20:29:38.475985 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:38.475999 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:38.476087 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:38.476347 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:38.476350 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:38.476666 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:38.476797 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:38.476804 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:38.477735 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:38.477739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:38.478591 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:38.478600 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:38.479218 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:38.479222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:38.479352 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:38.480023 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:38.480039 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:38.481972 (+ 1933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:38.484635 (+ 2663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:38.484642 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:38.484646 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:38.485018 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:38.485350 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:38.485356 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:38.485359 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:38.485490 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:38.485993 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:38.485998 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:38.486134 (+ 136us) client_negotiation.cc:769] Sending connection context
0929 20:29:38.486268 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 20:29:38.486548 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 20:29:38.486979 3872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:38.475667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34242 (local address 127.28.244.189:44795)
0929 20:29:38.475873 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:38.475881 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:38.476064 (+ 183us) server_negotiation.cc:407] Connection header received
0929 20:29:38.476172 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:38.476176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:38.476230 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:38.476304 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:38.476907 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:38.477616 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:38.478705 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:38.479088 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:38.479181 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:38.482146 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:38.482150 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:38.482153 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:38.482179 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:38.484506 (+ 2327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:38.485130 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:38.485137 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:38.485141 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:38.485249 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:38.485592 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:38.485599 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:38.485602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:38.485870 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:38.486019 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:38.486339 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 20:29:38.486675 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":148,"server-negotiator.queue_time_us":62}
I20260929 20:29:38.491714 3869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c: Bootstrap starting.
I20260929 20:29:38.494776 3869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:38.495740 3869 log.cc:824] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c: Log is configured to *not* fsync() on all Append() calls
I20260929 20:29:38.498508 3869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c: No bootstrap required, opened a new log
I20260929 20:29:38.505539 3869 raft_consensus.cc:348] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } }
I20260929 20:29:38.505851 3869 raft_consensus.cc:374] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:38.505923 3869 raft_consensus.cc:729] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 985844716800427a89b43c509229630c, State: Initialized, Role: FOLLOWER
I20260929 20:29:38.506502 3869 consensus_queue.cc:260] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } }
I20260929 20:29:38.507283 3876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } } }
I20260929 20:29:38.507594 3876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:29:38.508388 3869 sys_catalog.cc:564] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [sys.catalog]: configured and running, proceeding with master startup.
W20260929 20:29:38.517364 3887 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 20:29:38.517554 3887 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 20:29:38.519858 3802 master_runner.cc:428] Detected that this master 985844716800427a89b43c509229630c is joining an existing cluster
I20260929 20:29:38.519974 3802 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.244.189:44795
I20260929 20:29:38.546599 3873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:38.524732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33346 (local address 127.28.244.190:39805)
0929 20:29:38.524910 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:38.524917 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:38.525928 (+ 1011us) server_negotiation.cc:407] Connection header received
0929 20:29:38.526260 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:38.526270 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:38.526403 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:38.526503 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:38.528721 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:38.530249 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:38.533387 (+ 3138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:38.534342 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:38.534471 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:38.537967 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:38.537975 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:38.537980 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:38.538025 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:38.543115 (+ 5090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:38.544048 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:38.544054 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:38.544058 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:38.544214 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:38.544791 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:38.544796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:38.544800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:38.545167 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:38.545336 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:38.545972 (+ 636us) server_negotiation.cc:299] Negotiation successful
0929 20:29:38.546228 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:29:38.547708 3893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:38.525031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.0.0.1:33346)
0929 20:29:38.525805 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:38.525831 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:38.526050 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:38.526818 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:38.526826 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:38.527892 (+ 1066us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:38.528486 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:38.528504 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:38.530504 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:38.530513 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:38.533134 (+ 2621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:38.533157 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:38.534952 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:38.534963 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:38.535248 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:38.536093 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:38.536127 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:38.537722 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:38.543326 (+ 5604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:38.543333 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:38.543339 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:38.543906 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:38.544372 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:38.544377 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:38.544381 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:38.544650 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:38.545428 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:38.545433 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:38.545653 (+ 220us) client_negotiation.cc:769] Sending connection context
0929 20:29:38.546811 (+ 1158us) client_negotiation.cc:240] Negotiation successful
0929 20:29:38.547200 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":456,"threads_started":1}
WARNING: no policy specified for oryx/127.28.244.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.244.188@KRBTEST.COM" created.
I20260929 20:29:38.560902 3872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:38.524547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34254 (local address 127.28.244.189:44795)
0929 20:29:38.524770 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:38.524777 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:38.543712 (+ 18935us) server_negotiation.cc:407] Connection header received
0929 20:29:38.543898 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:38.543906 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:38.543994 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:38.544091 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:38.545582 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:38.546748 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:38.549871 (+ 3123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:38.550690 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:38.550816 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:38.553503 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:38.553514 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:38.553520 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:38.553559 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:38.557601 (+ 4042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:38.558449 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:38.558464 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:38.558471 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:38.558620 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:38.559032 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:38.559037 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:38.559040 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:38.559422 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:38.559587 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:38.560259 (+ 672us) server_negotiation.cc:299] Negotiation successful
0929 20:29:38.560509 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:29:38.562232 3894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:38.525992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:44795 (local address 127.0.0.1:34254)
0929 20:29:38.543591 (+ 17599us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:38.543611 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:38.543746 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:38.544269 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:38.544277 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:38.545032 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:38.545427 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:38.545443 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:38.546972 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:38.546977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:38.549703 (+ 2726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:38.549722 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:38.551165 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:38.551174 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:38.551411 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:38.552050 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:38.552083 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:38.553311 (+ 1228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:38.557734 (+ 4423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:38.557740 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:38.557744 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:38.558322 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:38.558743 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:38.558748 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:38.558752 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:38.558912 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:38.559663 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:38.559667 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:38.559912 (+ 245us) client_negotiation.cc:769] Sending connection context
0929 20:29:38.561482 (+ 1570us) client_negotiation.cc:240] Negotiation successful
0929 20:29:38.561772 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17448,"thread_start_us":17232,"threads_started":1}
Sep 29 20:29:38 dist-test-slave-p29l krb5kdc[3296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713776, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.244.190@KRBTEST.COM for oryx/127.28.244.189@KRBTEST.COM
I20260929 20:29:38.576144 3896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:38.555827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34260 (local address 127.28.244.189:44795)
0929 20:29:38.556556 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:38.556562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:38.557226 (+ 664us) server_negotiation.cc:407] Connection header received
0929 20:29:38.557632 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:38.557638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:38.557725 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:38.557839 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:38.559440 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:38.560929 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:38.563047 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:38.563876 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:38.563992 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:38.568697 (+ 4705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:38.568704 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:38.568709 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:38.568746 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:38.572581 (+ 3835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:38.573560 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:38.573567 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:38.573571 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:38.573738 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:38.574198 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:38.574203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:38.574206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:38.574585 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:38.574726 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:38.575375 (+ 649us) server_negotiation.cc:299] Negotiation successful
0929 20:29:38.575645 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":499,"threads_started":1}
I20260929 20:29:38.576207 3897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:38.556226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:44795 (local address 127.0.0.1:34260)
0929 20:29:38.557083 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:38.557137 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:38.557389 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:38.557903 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:38.557911 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:38.558894 (+ 983us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:38.559232 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:38.559248 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:38.561122 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:38.561130 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:38.562866 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:38.562886 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:38.564470 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:38.564478 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:38.564707 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:38.565492 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:38.565525 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:38.568445 (+ 2920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:38.572747 (+ 4302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:38.572764 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:38.572776 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:38.573419 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:38.573885 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:38.573893 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:38.573899 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:38.574074 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:38.574752 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:38.574764 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:38.575029 (+ 265us) client_negotiation.cc:769] Sending connection context
0929 20:29:38.575253 (+ 224us) client_negotiation.cc:240] Negotiation successful
0929 20:29:38.575671 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":453,"threads_started":1}
I20260929 20:29:38.578761 3340 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.244.189:44795: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "985844716800427a89b43c509229630c"
member_type: NON_VOTER
last_known_addr {
host: "127.28.244.189"
port: 44795
}
attrs {
promote: true
}
}
dest_uuid: "c23310ae5174458f8743b50aefe320bc"
cas_config_opid_index: -1
I20260929 20:29:38.580034 3340 consensus_queue.cc:237] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: NON_VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: true } }
I20260929 20:29:38.583377 3901 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 985844716800427a89b43c509229630c (127.28.244.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: NON_VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: true } } }
I20260929 20:29:38.584667 3844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:38.588238 3901 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.244.189:44795
I20260929 20:29:38.588438 3844 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [term 1 FOLLOWER]: Refusing update from remote peer c23310ae5174458f8743b50aefe320bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 20:29:38.589004 3802 master.cc:537] Master@127.28.244.189:44795 shutting down...
I20260929 20:29:38.588665 3900 sys_catalog.cc:455] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c23310ae5174458f8743b50aefe320bc" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: NON_VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: true } } }
I20260929 20:29:38.589502 3900 sys_catalog.cc:458] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.244.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.188.keytab.
Entry for principal oryx/127.28.244.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.188.keytab.
I20260929 20:29:38.593078 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.188:38887
--webserver_interface=127.28.244.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.188.keytab
--principal=oryx/127.28.244.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:39805,127.28.244.189:44795,127.28.244.188:38887
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:29:38.600499 3901 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "985844716800427a89b43c509229630c" member_type: NON_VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 20:29:38.603708 3328 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260929 20:29:38.606573 3802 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:29:38.606988 3802 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:29:38.607172 3802 tablet_replica.cc:330] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c: stopping tablet replica
I20260929 20:29:38.614522 3802 master.cc:552] Master@127.28.244.189:44795 shutdown complete.
I20260929 20:29:38.614688 3802 master_runner.cc:305] Clearing existing system tablet
I20260929 20:29:38.615211 3802 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:38.615391 3802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:38.621488 3897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:38.620957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:44795 (local address 127.0.0.1:34262)
0929 20:29:38.621128 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:38.621270 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.189:44795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 20:29:38.621877 3328 consensus_peers.cc:487] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc -> Peer 985844716800427a89b43c509229630c (127.28.244.189:44795): Couldn't send request to peer 985844716800427a89b43c509229630c. Status: Network error: Client connection negotiation failed: client connection to 127.28.244.189:44795: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 20:29:38.622789 3802 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 20:29:38.624928 3906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:38.625301 3802 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:38.625818 3802 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 20:29:38.625957 3802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-1/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-1/wal
uuid: "985844716800427a89b43c509229630c"
format_stamp: "Formatted at 2026-09-29 20:29:38 on dist-test-slave-p29l"
server_key: "ec1ce62dfbe74765c4267aa54241d54f"
server_key_iv: "a11981dd1b4258fa59fce9e1be3dbf06"
server_key_version: "encryptionkey@0"
I20260929 20:29:38.631994 3802 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:29:38.639497 3802 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 20:29:38.639870 3802 log.cc:1196] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 20:29:38.640439 3802 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c: Deleting consensus metadata
I20260929 20:29:38.640760 3802 master_runner.cc:315] Copying system tablet from 127.28.244.190:39805
I20260929 20:29:38.643407 3802 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c: tablet copy: Beginning tablet copy session from remote peer at address 127.28.244.190:39805
I20260929 20:29:38.663399 3873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:38.644390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33362 (local address 127.28.244.190:39805)
0929 20:29:38.644576 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:38.644583 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:38.648389 (+ 3806us) server_negotiation.cc:407] Connection header received
0929 20:29:38.648624 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:38.648630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:38.648738 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:38.648864 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:38.649898 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:38.651046 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:38.652848 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:38.653530 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:38.653666 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:38.656195 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:38.656202 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:38.656208 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:38.656249 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:38.659892 (+ 3643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:38.660864 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:38.660870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:38.660874 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:38.661034 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:38.661493 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:38.661518 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:38.661523 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:38.661912 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:38.662067 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:38.662758 (+ 691us) server_negotiation.cc:299] Negotiation successful
0929 20:29:38.663019 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:29:38.664214 3912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:38.644250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.0.0.1:33362)
0929 20:29:38.648226 (+ 3976us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:38.648272 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:38.648456 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:38.648920 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:38.648925 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:38.649457 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:38.649706 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:38.649718 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:38.651205 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:38.651210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:38.652678 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:38.652693 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:38.653724 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:38.653734 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:38.653961 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:38.654717 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:38.654740 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:38.655983 (+ 1243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:38.660066 (+ 4083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:38.660074 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:38.660080 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:38.660713 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:38.661157 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:38.661162 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:38.661165 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:38.661353 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:38.662121 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:38.662129 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:38.662409 (+ 280us) client_negotiation.cc:769] Sending connection context
0929 20:29:38.662599 (+ 190us) client_negotiation.cc:240] Negotiation successful
0929 20:29:38.662999 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3829,"thread_start_us":3706,"threads_started":1}
I20260929 20:29:38.665552 3361 tablet_copy_service.cc:140] P c23310ae5174458f8743b50aefe320bc: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 985844716800427a89b43c509229630c ({username='oryx', principal='oryx/127.28.244.189@KRBTEST.COM'} at 127.0.0.1:33362)
I20260929 20:29:38.665853 3361 tablet_copy_service.cc:161] P c23310ae5174458f8743b50aefe320bc: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 985844716800427a89b43c509229630c at {username='oryx', principal='oryx/127.28.244.189@KRBTEST.COM'} at 127.0.0.1:33362: session id = 985844716800427a89b43c509229630c-00000000000000000000000000000000
I20260929 20:29:38.670548 3361 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 20:29:38.672989 3802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 20:29:38.680042 3802 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c: tablet copy: Starting download of 0 data blocks...
I20260929 20:29:38.680370 3802 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c: tablet copy: Starting download of 1 WAL segments...
I20260929 20:29:38.683305 3802 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 20:29:38.688066 3361 tablet_copy_service.cc:342] P c23310ae5174458f8743b50aefe320bc: Request end of tablet copy session 985844716800427a89b43c509229630c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.244.189@KRBTEST.COM'} at 127.0.0.1:33362
I20260929 20:29:38.688299 3361 tablet_copy_service.cc:433] P c23310ae5174458f8743b50aefe320bc: ending tablet copy session 985844716800427a89b43c509229630c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 985844716800427a89b43c509229630c
I20260929 20:29:38.701565 3802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:38.704457 3913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:38.705610 3914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:38.713405 3916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:38.719457 3802 server_base.cc:1034] running on GCE node
I20260929 20:29:38.719875 3802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:38.720606 3802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:38.721901 3802 hybrid_clock.cc:648] HybridClock initialized: now 1790713778721792 us; error 67 us; skew 500 ppm
I20260929 20:29:38.722198 3802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:38 dist-test-slave-p29l krb5kdc[3296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713778, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.244.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:38.724522 3802 init.cc:377] Logged in from keytab as oryx/127.28.244.189@KRBTEST.COM (short username oryx)
I20260929 20:29:38.728842 3802 webserver.cc:460] Webserver started at http://127.28.244.189:46657/ using document root <none> and password file <none>
I20260929 20:29:38.729241 3802 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:38.729338 3802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:38.732894 3802 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 20:29:38.734990 3923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:38.735464 3802 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 20:29:38.735633 3802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-1/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-1/wal
uuid: "985844716800427a89b43c509229630c"
format_stamp: "Formatted at 2026-09-29 20:29:38 on dist-test-slave-p29l"
server_key: "ec1ce62dfbe74765c4267aa54241d54f"
server_key_iv: "a11981dd1b4258fa59fce9e1be3dbf06"
server_key_version: "encryptionkey@0"
I20260929 20:29:38.735878 3802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:38.759400 3802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:38.764288 3802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:38.789731 3802 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.189:44795
I20260929 20:29:38.789780 3974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.189:44795 every 8 connection(s)
I20260929 20:29:38.790969 3802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-1/data/info.pb
I20260929 20:29:38.796543 3975 sys_catalog.cc:263] Verifying existing consensus state
I20260929 20:29:38.800247 3975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c: Bootstrap starting.
I20260929 20:29:38.816481 3975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:29:38.817046 3975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c: Bootstrap complete.
I20260929 20:29:38.818096 3975 raft_consensus.cc:348] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: NON_VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: true } }
I20260929 20:29:38.818368 3975 raft_consensus.cc:729] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 985844716800427a89b43c509229630c, State: Initialized, Role: LEARNER
I20260929 20:29:38.818629 3975 consensus_queue.cc:260] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: NON_VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: true } }
I20260929 20:29:38.819845 3975 sys_catalog.cc:564] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:29:38.819700 3978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: NON_VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: true } } }
I20260929 20:29:38.820416 3978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [sys.catalog]: This master's current role is: LEARNER
I20260929 20:29:38.829079 3989 catalog_manager.cc:1259] Loaded cluster ID: bb52180e1176467eb311d00f60c824a6
I20260929 20:29:38.829190 3989 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c: loading cluster ID for follower catalog manager: success
I20260929 20:29:38.832891 3989 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c: acquiring CA information for follower catalog manager: success
I20260929 20:29:38.835632 3989 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 20:29:38.947556 3797 heartbeater.cc:503] Master 127.28.244.190:39805 was elected leader, sending a full tablet report...
W20260929 20:29:39.009706 3903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:39.010075 3903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:39.010226 3903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:39.018033 3903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:29:39.018193 3903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:39.018301 3903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:39.018409 3903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:29:39.018486 3903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:29:39.029857 3903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:39805,127.28.244.189:44795,127.28.244.188:38887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.188.keytab
--principal=oryx/127.28.244.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.188:38887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.244.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:39.031391 3903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:39.032823 3903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:39.045802 3994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:39.045780 3995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:39.045737 3997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:39.049432 3903 server_base.cc:1034] running on GCE node
I20260929 20:29:39.050289 3903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:39.051990 3903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:39.053236 3903 hybrid_clock.cc:648] HybridClock initialized: now 1790713779053173 us; error 80 us; skew 500 ppm
I20260929 20:29:39.053629 3903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:39 dist-test-slave-p29l krb5kdc[3296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713779, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.244.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:39.057123 3903 init.cc:377] Logged in from keytab as oryx/127.28.244.188@KRBTEST.COM (short username oryx)
I20260929 20:29:39.058956 3903 webserver.cc:460] Webserver started at http://127.28.244.188:38999/ using document root <none> and password file <none>
I20260929 20:29:39.059690 3903 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:39.059791 3903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:39.060199 3903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:39.063381 3903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-2/data/instance:
uuid: "7bfc4052f1a84dd184643c71ac194b23"
format_stamp: "Formatted at 2026-09-29 20:29:39 on dist-test-slave-p29l"
server_key: "04c142dcdb80a71fa7b8f6dbfba50c1f"
server_key_iv: "c0f35524a6102bdb495cbfc1ce5dfff5"
server_key_version: "encryptionkey@0"
I20260929 20:29:39.064317 3903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-2/wal/instance:
uuid: "7bfc4052f1a84dd184643c71ac194b23"
format_stamp: "Formatted at 2026-09-29 20:29:39 on dist-test-slave-p29l"
server_key: "04c142dcdb80a71fa7b8f6dbfba50c1f"
server_key_iv: "c0f35524a6102bdb495cbfc1ce5dfff5"
server_key_version: "encryptionkey@0"
I20260929 20:29:39.070083 3903 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260929 20:29:39.074108 4004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:39.075686 3903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:29:39.075927 3903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-2/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-2/wal
uuid: "7bfc4052f1a84dd184643c71ac194b23"
format_stamp: "Formatted at 2026-09-29 20:29:39 on dist-test-slave-p29l"
server_key: "04c142dcdb80a71fa7b8f6dbfba50c1f"
server_key_iv: "c0f35524a6102bdb495cbfc1ce5dfff5"
server_key_version: "encryptionkey@0"
I20260929 20:29:39.076161 3903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:39.105551 3903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:39.110052 3903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:39.110376 3903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:39.129851 3903 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.188:38887
I20260929 20:29:39.129874 4055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.188:38887 every 8 connection(s)
I20260929 20:29:39.131397 3903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-2/data/info.pb
I20260929 20:29:39.135486 4056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:39.136145 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 3903
I20260929 20:29:39.145299 4056 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:29:39 dist-test-slave-p29l krb5kdc[3296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.244.188@KRBTEST.COM
Sep 29 20:29:39 dist-test-slave-p29l krb5kdc[3296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713779, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.244.188@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM
I20260929 20:29:39.161010 4059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:39.138271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49820 (local address 127.28.244.188:38887)
0929 20:29:39.139178 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:39.139190 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:39.140336 (+ 1146us) server_negotiation.cc:407] Connection header received
0929 20:29:39.142419 (+ 2083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:39.142444 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:39.142882 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:39.143338 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:39.144921 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.146386 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.147781 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.148780 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.148990 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:39.153295 (+ 4305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:39.153303 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:39.153313 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:39.153351 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:39.156788 (+ 3437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:39.157444 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:39.157450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:39.157458 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:39.157640 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:39.158127 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:39.158130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:39.158132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:39.158734 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:39.159006 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:39.159470 (+ 464us) server_negotiation.cc:299] Negotiation successful
0929 20:29:39.159790 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":417,"threads_started":1}
I20260929 20:29:39.163211 3873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:39.148311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33366 (local address 127.28.244.190:39805)
0929 20:29:39.148517 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:39.148524 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:39.149247 (+ 723us) server_negotiation.cc:407] Connection header received
0929 20:29:39.149517 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:39.149524 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:39.149603 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:39.149709 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:39.150811 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.151494 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.153004 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.153394 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.153540 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:39.157070 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:39.157074 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:39.157077 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:39.157110 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:39.160195 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:39.161006 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:39.161013 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:39.161016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:39.161219 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:39.161695 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:39.161701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:39.161707 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:39.162062 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:39.162213 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:39.162647 (+ 434us) server_negotiation.cc:299] Negotiation successful
0929 20:29:39.162875 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:29:39.163311 4060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:39.148231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.0.0.1:33366)
0929 20:29:39.149108 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:39.149161 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:39.149370 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:39.149771 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:39.149779 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:39.150408 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:39.150657 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:39.150666 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.151641 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.151646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:39.152862 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:39.152874 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.153580 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.153586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:39.153784 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:39.154412 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:39.154484 (+ 72us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:39.156854 (+ 2370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:39.160325 (+ 3471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:39.160338 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:39.160345 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:39.160877 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:39.161385 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:39.161388 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:39.161390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:39.161581 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:39.162189 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:39.162194 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:39.162335 (+ 141us) client_negotiation.cc:769] Sending connection context
0929 20:29:39.162542 (+ 207us) client_negotiation.cc:240] Negotiation successful
0929 20:29:39.162868 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":473,"threads_started":1}
I20260929 20:29:39.164698 4056 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:29:39 dist-test-slave-p29l krb5kdc[3296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713779, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.244.188@KRBTEST.COM for oryx/127.28.244.189@KRBTEST.COM
I20260929 20:29:39.177812 4060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:39.165481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:44795 (local address 127.0.0.1:34268)
0929 20:29:39.165781 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:39.165805 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:39.165928 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:39.166375 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:39.166378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:39.166714 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:39.166848 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:39.166856 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.168231 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.168235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:39.169310 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:39.169322 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.169980 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.169984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:39.170098 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:39.170687 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:39.170704 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:39.172713 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:39.175726 (+ 3013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:39.175730 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:39.175733 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:39.176161 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:39.176515 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:39.176518 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:39.176520 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:39.176637 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:39.177102 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:39.177106 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:39.177227 (+ 121us) client_negotiation.cc:769] Sending connection context
0929 20:29:39.177334 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 20:29:39.177579 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260929 20:29:39.177831 4061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:39.165504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34268 (local address 127.28.244.189:44795)
0929 20:29:39.166126 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:39.166130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:39.166145 (+ 15us) server_negotiation.cc:407] Connection header received
0929 20:29:39.166193 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:39.166197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:39.166252 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:39.166339 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:39.166990 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.168123 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.169434 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.169829 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.169932 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:39.172923 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:39.172927 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:39.172931 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:39.172962 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:39.175595 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:39.176271 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:39.176275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:39.176278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:39.176415 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:39.176743 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:39.176746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:39.176748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:39.177005 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:39.177123 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:39.177417 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 20:29:39.177587 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":258,"threads_started":1}
I20260929 20:29:39.178947 4056 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.244.188" port: 38887 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:29:39 dist-test-slave-p29l krb5kdc[3296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713779, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.244.188@KRBTEST.COM for oryx/127.28.244.188@KRBTEST.COM
I20260929 20:29:39.191378 4059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:39.179764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49824 (local address 127.28.244.188:38887)
0929 20:29:39.179922 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:39.179929 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:39.179947 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:29:39.180105 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:39.180110 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:39.180169 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:39.180262 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:39.180930 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.181676 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.182736 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.183142 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.183281 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:39.186149 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:39.186154 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:39.186159 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:39.186196 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:39.188820 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:39.189492 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:39.189497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:39.189499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:39.189598 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:39.189954 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:39.189959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:39.189962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:39.190240 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:39.190391 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:39.190737 (+ 346us) server_negotiation.cc:299] Negotiation successful
0929 20:29:39.191070 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":133,"server-negotiator.queue_time_us":57}
I20260929 20:29:39.191380 4060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:39.179666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:38887 (local address 127.0.0.1:49824)
0929 20:29:39.179884 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:39.179902 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:39.180002 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:39.180276 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:39.180280 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:39.180652 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:39.180807 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:39.180814 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.181794 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.181801 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:39.182607 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:39.182617 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.183297 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.183303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:39.183488 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:39.184120 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:39.184137 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:39.185985 (+ 1848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:39.188952 (+ 2967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:39.188979 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:39.188983 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:39.189340 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:39.189701 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:39.189708 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:39.189712 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:39.189855 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:39.190353 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:39.190357 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:39.190520 (+ 163us) client_negotiation.cc:769] Sending connection context
0929 20:29:39.190656 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 20:29:39.190935 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 20:29:39.196523 4056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23: Bootstrap starting.
I20260929 20:29:39.199738 4056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:39.200871 4056 log.cc:824] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23: Log is configured to *not* fsync() on all Append() calls
I20260929 20:29:39.203504 4056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23: No bootstrap required, opened a new log
I20260929 20:29:39.210649 4056 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } } peers { permanent_uuid: "7bfc4052f1a84dd184643c71ac194b23" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 38887 } }
I20260929 20:29:39.210980 4056 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:39.211124 4056 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bfc4052f1a84dd184643c71ac194b23, State: Initialized, Role: FOLLOWER
I20260929 20:29:39.211782 4056 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } } peers { permanent_uuid: "7bfc4052f1a84dd184643c71ac194b23" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 38887 } }
I20260929 20:29:39.212777 4063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } } peers { permanent_uuid: "7bfc4052f1a84dd184643c71ac194b23" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 38887 } } }
I20260929 20:29:39.213090 4063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:29:39.213783 4056 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:29:39.218868 4075 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 20:29:39.225623 4061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:39.218830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34272 (local address 127.28.244.189:44795)
0929 20:29:39.219043 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:39.219051 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:39.219241 (+ 190us) server_negotiation.cc:407] Connection header received
0929 20:29:39.219490 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:39.219499 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:39.219633 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:39.219781 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:29:39.220256 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.221329 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.223315 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.224614 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.224785 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:39.224991 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:39.225197 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 20:29:39.225329 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 20:29:39.225656 4076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:39.218630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:44795 (local address 127.0.0.1:34272)
0929 20:29:39.219110 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:39.219149 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:39.219296 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:39.219846 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:39.219851 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:39.219883 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:29:39.220108 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:39.220118 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.221459 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.221465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:39.223164 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:39.223177 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.224791 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.224797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:39.224989 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:39.225054 (+ 65us) client_negotiation.cc:769] Sending connection context
0929 20:29:39.225173 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 20:29:39.225318 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":237,"threads_started":1}
I20260929 20:29:39.227080 3950 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260929 20:29:39.227700 3978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c23310ae5174458f8743b50aefe320bc. Latest consensus state: current_term: 1 leader_uuid: "c23310ae5174458f8743b50aefe320bc" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: NON_VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: true } } }
I20260929 20:29:39.227890 3903 master_runner.cc:428] Detected that this master 7bfc4052f1a84dd184643c71ac194b23 is joining an existing cluster
I20260929 20:29:39.227993 3978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [sys.catalog]: This master's current role is: LEARNER
I20260929 20:29:39.227994 3903 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.244.188:38887
I20260929 20:29:39.247531 4059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:39.232381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49830 (local address 127.28.244.188:38887)
0929 20:29:39.232819 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:39.232825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:39.233225 (+ 400us) server_negotiation.cc:407] Connection header received
0929 20:29:39.233456 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:39.233461 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:39.233563 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:39.233667 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:39.235024 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.235942 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.237516 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.237996 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.238136 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:39.240692 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:39.240703 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:39.240709 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:39.240747 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:39.244378 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:39.245177 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:39.245183 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:39.245186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:39.245346 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:39.245748 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:39.245754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:39.245758 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:39.246189 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:39.246414 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:39.246753 (+ 339us) server_negotiation.cc:299] Negotiation successful
0929 20:29:39.247171 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":152,"server-negotiator.queue_time_us":263}
I20260929 20:29:39.247560 4081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:39.232387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:38887 (local address 127.0.0.1:49830)
0929 20:29:39.233130 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:39.233153 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:39.233312 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:39.233865 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:39.233869 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:39.234550 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:39.234896 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:39.234906 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.236131 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.236136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:39.237379 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:39.237391 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.238155 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.238160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:39.238336 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:39.239071 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:39.239192 (+ 121us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:39.240501 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:39.244523 (+ 4022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:39.244528 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:39.244532 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:39.245050 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:39.245470 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:39.245475 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:39.245479 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:39.245627 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:39.246327 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:39.246331 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:39.246487 (+ 156us) client_negotiation.cc:769] Sending connection context
0929 20:29:39.246668 (+ 181us) client_negotiation.cc:240] Negotiation successful
0929 20:29:39.247010 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"mutex_wait_us":3,"thread_start_us":398,"threads_started":1}
I20260929 20:29:39.251214 4083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:39.232477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.0.0.1:33380)
0929 20:29:39.233924 (+ 1447us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:39.233944 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:39.234064 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:39.234950 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:39.234958 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:39.235916 (+ 958us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:39.236531 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:39.236549 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.238562 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.238570 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:39.240943 (+ 2373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:39.240953 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.241745 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.241754 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:39.241993 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:39.242609 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:39.242631 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:39.243919 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:39.248663 (+ 4744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:39.248667 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:39.248671 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:39.249088 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:39.249519 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:39.249522 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:39.249524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:39.249712 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:39.250458 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:39.250462 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:39.250607 (+ 145us) client_negotiation.cc:769] Sending connection context
0929 20:29:39.250709 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 20:29:39.250932 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1317,"mutex_wait_us":25,"thread_start_us":6852,"threads_started":1}
I20260929 20:29:39.251528 3873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:39.232108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33380 (local address 127.28.244.190:39805)
0929 20:29:39.232336 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:39.232343 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:39.234052 (+ 1709us) server_negotiation.cc:407] Connection header received
0929 20:29:39.234199 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:39.234208 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:39.234312 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:39.234461 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:39.236755 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.238243 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.241084 (+ 2841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.241543 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.241682 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:39.244130 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:39.244137 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:39.244143 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:39.244190 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:39.248515 (+ 4325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:39.249228 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:39.249235 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:39.249241 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:39.249394 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:39.249879 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:39.249886 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:39.249892 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:39.250309 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:39.250470 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:39.250929 (+ 459us) server_negotiation.cc:299] Negotiation successful
0929 20:29:39.251163 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:29:39.254127 4061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:39.232108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34288 (local address 127.28.244.189:44795)
0929 20:29:39.232621 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:39.232627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:39.239977 (+ 7350us) server_negotiation.cc:407] Connection header received
0929 20:29:39.240213 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:39.240220 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:39.240304 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:39.240392 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:39.241874 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.242911 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.244693 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.245369 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.245485 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:39.248165 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:39.248172 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:39.248177 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:39.248218 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:39.251435 (+ 3217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:39.252164 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:39.252169 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:39.252173 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:39.252319 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:39.252697 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:39.252701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:39.252705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:39.252970 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:39.253120 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:39.253555 (+ 435us) server_negotiation.cc:299] Negotiation successful
0929 20:29:39.253780 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377}
I20260929 20:29:39.254176 4082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:39.233265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:44795 (local address 127.0.0.1:34288)
0929 20:29:39.239881 (+ 6616us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:39.239899 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:39.240073 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:39.240595 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:39.240601 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:39.241338 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:39.241681 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:39.241694 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.243108 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.243112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:39.244545 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:39.244555 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.245687 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.245720 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:39.245910 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:39.246470 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:39.246493 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:39.247969 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:39.251594 (+ 3625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:39.251598 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:39.251601 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:39.252038 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:39.252434 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:39.252439 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:39.252443 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:39.252580 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:39.253102 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:39.253107 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:39.253294 (+ 187us) client_negotiation.cc:769] Sending connection context
0929 20:29:39.253430 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 20:29:39.253740 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6370,"thread_start_us":391,"threads_started":1}
Sep 29 20:29:39 dist-test-slave-p29l krb5kdc[3296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713776, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.244.190@KRBTEST.COM for oryx/127.28.244.188@KRBTEST.COM
I20260929 20:29:39.267489 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 3402
I20260929 20:29:39.269284 4059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:39.256463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49844 (local address 127.28.244.188:38887)
0929 20:29:39.256616 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:39.256620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:39.256680 (+ 60us) server_negotiation.cc:407] Connection header received
0929 20:29:39.256826 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:39.256830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:39.256942 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:39.257024 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:39.257757 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.258429 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.259551 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.260038 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.260133 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:39.263145 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:39.263149 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:39.263153 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:39.263183 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:39.266116 (+ 2933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:39.267100 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:39.267104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:39.267106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:39.267271 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:39.267717 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:39.267724 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:39.267727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:39.268169 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:39.268317 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:39.268778 (+ 461us) server_negotiation.cc:299] Negotiation successful
0929 20:29:39.268960 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 20:29:39.269647 4076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:39.256383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:38887 (local address 127.0.0.1:49844)
0929 20:29:39.256604 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:39.256620 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:39.256710 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:39.257089 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:39.257093 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:39.257454 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:39.257614 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:39.257623 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.258556 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.258560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:39.259409 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:39.259420 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.260160 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.260164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:39.260289 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:39.260900 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:39.260918 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:39.262949 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:39.266288 (+ 3339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:39.266294 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:39.266297 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:39.266828 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:39.267394 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:39.267398 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:39.267400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:39.267589 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:39.268303 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:39.268308 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:39.268500 (+ 192us) client_negotiation.cc:769] Sending connection context
0929 20:29:39.268668 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 20:29:39.269105 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 20:29:39.271138 3340 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.244.188:38887: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7bfc4052f1a84dd184643c71ac194b23"
member_type: NON_VOTER
last_known_addr {
host: "127.28.244.188"
port: 38887
}
attrs {
promote: true
}
}
dest_uuid: "c23310ae5174458f8743b50aefe320bc"
cas_config_opid_index: 5
I20260929 20:29:39.272291 3340 consensus_queue.cc:237] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: NON_VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: true } } peers { permanent_uuid: "7bfc4052f1a84dd184643c71ac194b23" member_type: NON_VOTER last_known_addr { host: "127.28.244.188" port: 38887 } attrs { promote: true } }
I20260929 20:29:39.274080 4069 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7bfc4052f1a84dd184643c71ac194b23 (127.28.244.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: NON_VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: true } } peers { permanent_uuid: "7bfc4052f1a84dd184643c71ac194b23" member_type: NON_VOTER last_known_addr { host: "127.28.244.188" port: 38887 } attrs { promote: true } } }
I20260929 20:29:39.274722 3950 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [term 1 LEARNER]: Refusing update from remote peer c23310ae5174458f8743b50aefe320bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 20:29:39.275460 4085 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "985844716800427a89b43c509229630c" member_type: NON_VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 20:29:39.277696 4069 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.244.188:38887
I20260929 20:29:39.278499 3903 master.cc:537] Master@127.28.244.188:38887 shutting down...
I20260929 20:29:39.278909 4031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:39.280799 3950 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7bfc4052f1a84dd184643c71ac194b23 (127.28.244.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: NON_VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: true } } peers { permanent_uuid: "7bfc4052f1a84dd184643c71ac194b23" member_type: NON_VOTER last_known_addr { host: "127.28.244.188" port: 38887 } attrs { promote: true } } }
I20260929 20:29:39.283567 4085 sys_catalog.cc:455] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c23310ae5174458f8743b50aefe320bc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: NON_VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: true } } peers { permanent_uuid: "7bfc4052f1a84dd184643c71ac194b23" member_type: NON_VOTER last_known_addr { host: "127.28.244.188" port: 38887 } attrs { promote: true } } }
I20260929 20:29:39.283950 4085 sys_catalog.cc:458] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [sys.catalog]: This master's current role is: LEADER
I20260929 20:29:39.285004 3978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c23310ae5174458f8743b50aefe320bc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: NON_VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: true } } peers { permanent_uuid: "7bfc4052f1a84dd184643c71ac194b23" member_type: NON_VOTER last_known_addr { host: "127.28.244.188" port: 38887 } attrs { promote: true } } }
I20260929 20:29:39.285313 3978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [sys.catalog]: This master's current role is: LEARNER
I20260929 20:29:39.288293 4069 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc: attempting to promote NON_VOTER 985844716800427a89b43c509229630c to VOTER
Sep 29 20:29:39 dist-test-slave-p29l krb5kdc[3296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.244.129@KRBTEST.COM
I20260929 20:29:39.289043 4069 consensus_queue.cc:237] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: false } } peers { permanent_uuid: "7bfc4052f1a84dd184643c71ac194b23" member_type: NON_VOTER last_known_addr { host: "127.28.244.188" port: 38887 } attrs { promote: true } }
I20260929 20:29:39.292029 3950 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [term 1 LEARNER]: Refusing update from remote peer c23310ae5174458f8743b50aefe320bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 20:29:39.292609 4085 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "985844716800427a89b43c509229630c" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 20:29:39.296406 4031 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [term 1 FOLLOWER]: Refusing update from remote peer c23310ae5174458f8743b50aefe320bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20260929 20:29:39.301736 3326 consensus_peers.cc:487] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc -> Peer 7bfc4052f1a84dd184643c71ac194b23 (127.28.244.188:38887): Couldn't send request to peer 7bfc4052f1a84dd184643c71ac194b23. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 20:29:39.308419 4084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:39.268536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34936 (local address 127.28.244.129:38387)
0929 20:29:39.270521 (+ 1985us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:39.270528 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:39.270552 (+ 24us) server_negotiation.cc:407] Connection header received
0929 20:29:39.270608 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:39.270615 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:39.270936 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:39.271087 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:39.272813 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.274352 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.283262 (+ 8910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.284341 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.284512 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:39.290695 (+ 6183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:39.290713 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:39.290733 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:39.290788 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:39.303303 (+ 12515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:39.304391 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:39.304406 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:39.304413 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:39.304639 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:39.305131 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:39.305137 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:39.305140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:39.305665 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:39.306014 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:39.307658 (+ 1644us) server_negotiation.cc:299] Negotiation successful
0929 20:29:39.308003 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1823,"thread_start_us":1665,"threads_started":1}
I20260929 20:29:39.309572 3903 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:29:39.309897 3525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:29:39.310027 3903 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:29:39.310276 3903 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23: stopping tablet replica
I20260929 20:29:39.311148 4086 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 985844716800427a89b43c509229630c (127.28.244.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: false } } peers { permanent_uuid: "7bfc4052f1a84dd184643c71ac194b23" member_type: NON_VOTER last_known_addr { host: "127.28.244.188" port: 38887 } attrs { promote: true } } }
I20260929 20:29:39.316327 4089 sys_catalog.cc:455] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c23310ae5174458f8743b50aefe320bc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: false } } peers { permanent_uuid: "7bfc4052f1a84dd184643c71ac194b23" member_type: NON_VOTER last_known_addr { host: "127.28.244.188" port: 38887 } attrs { promote: true } } }
I20260929 20:29:39.316790 4089 sys_catalog.cc:458] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [sys.catalog]: This master's current role is: LEADER
I20260929 20:29:39.317358 3903 master.cc:552] Master@127.28.244.188:38887 shutdown complete.
I20260929 20:29:39.317437 3903 master_runner.cc:305] Clearing existing system tablet
I20260929 20:29:39.316838 4086 sys_catalog.cc:455] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c23310ae5174458f8743b50aefe320bc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: false } } peers { permanent_uuid: "7bfc4052f1a84dd184643c71ac194b23" member_type: NON_VOTER last_known_addr { host: "127.28.244.188" port: 38887 } attrs { promote: true } } }
I20260929 20:29:39.317574 4086 sys_catalog.cc:458] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [sys.catalog]: This master's current role is: LEADER
I20260929 20:29:39.317830 3903 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:39.317909 3903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:39.317589 3950 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 985844716800427a89b43c509229630c (127.28.244.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: false } } peers { permanent_uuid: "7bfc4052f1a84dd184643c71ac194b23" member_type: NON_VOTER last_known_addr { host: "127.28.244.188" port: 38887 } attrs { promote: true } } }
I20260929 20:29:39.320258 3903 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 20:29:39.321769 4093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:39.322381 3903 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:39.322342 3978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c23310ae5174458f8743b50aefe320bc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: false } } peers { permanent_uuid: "7bfc4052f1a84dd184643c71ac194b23" member_type: NON_VOTER last_known_addr { host: "127.28.244.188" port: 38887 } attrs { promote: true } } }
I20260929 20:29:39.322635 3978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:29:39.322655 3903 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 20:29:39.322762 3903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-2/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-2/wal
uuid: "7bfc4052f1a84dd184643c71ac194b23"
format_stamp: "Formatted at 2026-09-29 20:29:39 on dist-test-slave-p29l"
server_key: "04c142dcdb80a71fa7b8f6dbfba50c1f"
server_key_iv: "c0f35524a6102bdb495cbfc1ce5dfff5"
server_key_version: "encryptionkey@0"
I20260929 20:29:39.326608 3903 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:29:39.331540 3903 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 20:29:39.331701 3903 log.cc:1196] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 20:29:39.332103 3903 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23: Deleting consensus metadata
I20260929 20:29:39.332367 3903 master_runner.cc:315] Copying system tablet from 127.28.244.190:39805
I20260929 20:29:39.334800 3903 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23: tablet copy: Beginning tablet copy session from remote peer at address 127.28.244.190:39805
I20260929 20:29:39.347543 3873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:39.335639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33394 (local address 127.28.244.190:39805)
0929 20:29:39.335942 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:39.335949 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:39.336171 (+ 222us) server_negotiation.cc:407] Connection header received
0929 20:29:39.336396 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:39.336402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:39.336487 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:39.336601 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:39.337358 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.338144 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.339363 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.339780 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.339965 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:39.341800 (+ 1835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:39.341806 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:39.341811 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:39.341852 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:39.344874 (+ 3022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:39.345505 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:39.345512 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:39.345516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:39.345651 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:39.346045 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:39.346051 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:39.346054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:39.346343 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:39.346524 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:39.346926 (+ 402us) server_negotiation.cc:299] Negotiation successful
0929 20:29:39.347178 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260929 20:29:39.347632 4099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:39.335548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.0.0.1:33394)
0929 20:29:39.336061 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:39.336080 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:39.336245 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:39.336627 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:39.336632 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:39.337027 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:39.337200 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:39.337209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.338299 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.338305 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:39.339204 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:39.339242 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.339977 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.339983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:39.340149 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:39.340827 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:39.340852 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:39.341602 (+ 750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:39.345019 (+ 3417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:39.345025 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:39.345029 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:39.345381 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:39.345766 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:39.345771 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:39.345775 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:39.345933 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:39.346474 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:39.346481 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:39.346653 (+ 172us) client_negotiation.cc:769] Sending connection context
0929 20:29:39.346799 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 20:29:39.347165 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":261,"threads_started":1}
I20260929 20:29:39.348572 3361 tablet_copy_service.cc:140] P c23310ae5174458f8743b50aefe320bc: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7bfc4052f1a84dd184643c71ac194b23 ({username='oryx', principal='oryx/127.28.244.188@KRBTEST.COM'} at 127.0.0.1:33394)
I20260929 20:29:39.348737 3361 tablet_copy_service.cc:161] P c23310ae5174458f8743b50aefe320bc: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7bfc4052f1a84dd184643c71ac194b23 at {username='oryx', principal='oryx/127.28.244.188@KRBTEST.COM'} at 127.0.0.1:33394: session id = 7bfc4052f1a84dd184643c71ac194b23-00000000000000000000000000000000
I20260929 20:29:39.351046 3361 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 20:29:39.352830 3903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 20:29:39.357321 3903 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23: tablet copy: Starting download of 0 data blocks...
I20260929 20:29:39.357610 3903 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23: tablet copy: Starting download of 1 WAL segments...
I20260929 20:29:39.360002 3903 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 20:29:39.363293 3361 tablet_copy_service.cc:342] P c23310ae5174458f8743b50aefe320bc: Request end of tablet copy session 7bfc4052f1a84dd184643c71ac194b23-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.244.188@KRBTEST.COM'} at 127.0.0.1:33394
I20260929 20:29:39.363412 3361 tablet_copy_service.cc:433] P c23310ae5174458f8743b50aefe320bc: ending tablet copy session 7bfc4052f1a84dd184643c71ac194b23-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7bfc4052f1a84dd184643c71ac194b23
I20260929 20:29:39.372655 3903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:39.375048 4100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:39.375244 4101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:39.379345 3903 server_base.cc:1034] running on GCE node
W20260929 20:29:39.379632 4103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:39.380062 3903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:39.380807 3903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:39.381994 3903 hybrid_clock.cc:648] HybridClock initialized: now 1790713779381949 us; error 45 us; skew 500 ppm
I20260929 20:29:39.382218 3903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:39 dist-test-slave-p29l krb5kdc[3296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713779, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.244.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:39.384263 3903 init.cc:377] Logged in from keytab as oryx/127.28.244.188@KRBTEST.COM (short username oryx)
I20260929 20:29:39.385510 3903 webserver.cc:460] Webserver started at http://127.28.244.188:43817/ using document root <none> and password file <none>
I20260929 20:29:39.385799 3903 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:39.385901 3903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:39.388144 3903 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 20:29:39.389695 4110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:39.390084 3903 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 20:29:39.390221 3903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-2/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-2/wal
uuid: "7bfc4052f1a84dd184643c71ac194b23"
format_stamp: "Formatted at 2026-09-29 20:29:39 on dist-test-slave-p29l"
server_key: "04c142dcdb80a71fa7b8f6dbfba50c1f"
server_key_iv: "c0f35524a6102bdb495cbfc1ce5dfff5"
server_key_version: "encryptionkey@0"
I20260929 20:29:39.390405 3903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:39.402791 3903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:39.405606 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 3402
I20260929 20:29:39.406765 3903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:39.426872 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.129:38387
--local_ip_for_outbound_sockets=127.28.244.129
--tserver_master_addrs=127.28.244.190:39805
--webserver_port=35433
--webserver_interface=127.28.244.129
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.129.keytab
--principal=oryx/127.28.244.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.244.190:39805,127.28.244.189:44795
--tserver_master_addrs=127.28.244.190:39805,127.28.244.189:44795,127.28.244.188:38887 with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:29:39.427731 3903 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.188:38887
I20260929 20:29:39.427765 4161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.188:38887 every 8 connection(s)
I20260929 20:29:39.428642 3903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/master-2/data/info.pb
I20260929 20:29:39.432269 4162 sys_catalog.cc:263] Verifying existing consensus state
I20260929 20:29:39.435240 4162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23: Bootstrap starting.
I20260929 20:29:39.449440 4162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:29:39.449895 4162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23: Bootstrap complete.
I20260929 20:29:39.450738 4162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: false } } peers { permanent_uuid: "7bfc4052f1a84dd184643c71ac194b23" member_type: NON_VOTER last_known_addr { host: "127.28.244.188" port: 38887 } attrs { promote: true } }
I20260929 20:29:39.450953 4162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7bfc4052f1a84dd184643c71ac194b23, State: Initialized, Role: LEARNER
I20260929 20:29:39.451196 4162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: false } } peers { permanent_uuid: "7bfc4052f1a84dd184643c71ac194b23" member_type: NON_VOTER last_known_addr { host: "127.28.244.188" port: 38887 } attrs { promote: true } }
I20260929 20:29:39.451968 4166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: false } } peers { permanent_uuid: "7bfc4052f1a84dd184643c71ac194b23" member_type: NON_VOTER last_known_addr { host: "127.28.244.188" port: 38887 } attrs { promote: true } } }
I20260929 20:29:39.452278 4162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:29:39.452292 4166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [sys.catalog]: This master's current role is: LEARNER
I20260929 20:29:39.468179 4177 catalog_manager.cc:1259] Loaded cluster ID: bb52180e1176467eb311d00f60c824a6
I20260929 20:29:39.468304 4177 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23: loading cluster ID for follower catalog manager: success
I20260929 20:29:39.471750 4177 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23: acquiring CA information for follower catalog manager: success
I20260929 20:29:39.474457 4177 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 20:29:39.782590 4164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:39.783013 4164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:39.783155 4164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:39.796484 4164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:39.796666 4164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:39.796934 4164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.129
I20260929 20:29:39.813580 4164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.129.keytab
--principal=oryx/127.28.244.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.129:38387
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.244.129
--webserver_port=35433
--tserver_master_addrs=127.28.244.190:39805,127.28.244.189:44795,127.28.244.188:38887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.129
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:39.815254 4164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:39.817219 4164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 20:29:39.820400 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:39.792033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:38887 (local address 127.0.0.1:49852)
0929 20:29:39.807432 (+ 15399us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:39.807457 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:39.807628 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:39.812988 (+ 5360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:39.812994 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:39.813028 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:29:39.813222 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:39.813234 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.815359 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.815368 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:39.817333 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:39.817353 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.819429 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.819439 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:39.819619 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:39.819652 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 20:29:39.819921 (+ 269us) client_negotiation.cc:240] Negotiation successful
0929 20:29:39.820076 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15240,"thread_start_us":15693,"threads_started":1}
I20260929 20:29:39.821478 4183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:39.799008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49852 (local address 127.28.244.188:38887)
0929 20:29:39.812352 (+ 13344us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:39.812358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:39.812384 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:29:39.812466 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:39.812471 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:39.812574 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:39.812717 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:29:39.813444 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.815149 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.817566 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.818871 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.819071 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:39.819263 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:39.820883 (+ 1620us) server_negotiation.cc:299] Negotiation successful
0929 20:29:39.821035 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13176,"thread_start_us":12996,"threads_started":1}
I20260929 20:29:39.822542 4137 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260929 20:29:39.823202 4166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c23310ae5174458f8743b50aefe320bc. Latest consensus state: current_term: 1 leader_uuid: "c23310ae5174458f8743b50aefe320bc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: false } } peers { permanent_uuid: "7bfc4052f1a84dd184643c71ac194b23" member_type: NON_VOTER last_known_addr { host: "127.28.244.188" port: 38887 } attrs { promote: true } } }
I20260929 20:29:39.823527 4166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [sys.catalog]: This master's current role is: LEARNER
W20260929 20:29:39.836799 4185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:39.837213 4186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:39.838837 4188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:39.839282 4164 server_base.cc:1034] running on GCE node
I20260929 20:29:39.840058 4164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:39.841114 4164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:39.842322 4164 hybrid_clock.cc:648] HybridClock initialized: now 1790713779842272 us; error 40 us; skew 500 ppm
I20260929 20:29:39.842682 4164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:39 dist-test-slave-p29l krb5kdc[3296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713779, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.244.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:39.845958 4164 init.cc:377] Logged in from keytab as oryx/127.28.244.129@KRBTEST.COM (short username oryx)
I20260929 20:29:39.847553 4164 webserver.cc:460] Webserver started at http://127.28.244.129:35433/ using document root <none> and password file <none>
I20260929 20:29:39.848376 4164 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:39.848539 4164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:39.854372 4164 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 20:29:39.857268 4195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:39.858901 4164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:29:39.859120 4164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-0/wal
uuid: "e9b791bada31439da862ec8bf5a97488"
format_stamp: "Formatted at 2026-09-29 20:29:36 on dist-test-slave-p29l"
server_key: "56eb302287518ab0b0042db4326f6e84"
server_key_iv: "f3b3802acfbe466ac1c963ccae87482f"
server_key_version: "encryptionkey@0"
I20260929 20:29:39.860404 4164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:39.893716 4164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:39.898392 4164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:39.898712 4164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:39.899734 4164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:29:39.901284 4164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:29:39.901376 4164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:39.901532 4164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:29:39.901611 4164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:39.937597 4164 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.129:38387
I20260929 20:29:39.937729 4307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.129:38387 every 8 connection(s)
I20260929 20:29:39.939098 4164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
I20260929 20:29:39.942646 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 4164
I20260929 20:29:39.942997 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 3536
Sep 29 20:29:39 dist-test-slave-p29l krb5kdc[3296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713779, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.244.129@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM
Sep 29 20:29:39 dist-test-slave-p29l krb5kdc[3296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713779, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.244.129@KRBTEST.COM for oryx/127.28.244.188@KRBTEST.COM
Sep 29 20:29:39 dist-test-slave-p29l krb5kdc[3296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.244.130@KRBTEST.COM
Sep 29 20:29:39 dist-test-slave-p29l krb5kdc[3296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713779, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.244.129@KRBTEST.COM for oryx/127.28.244.189@KRBTEST.COM
I20260929 20:29:39.986526 4183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:39.943746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:33657 (local address 127.28.244.188:38887)
0929 20:29:39.944001 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:39.944011 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:39.967961 (+ 23950us) server_negotiation.cc:407] Connection header received
0929 20:29:39.968122 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:39.968129 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:39.968220 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:39.968326 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:39.969775 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.970805 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.972970 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.973624 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.973719 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:39.978706 (+ 4987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:39.978712 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:39.978717 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:39.978763 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:39.982263 (+ 3500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:39.983360 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:39.983367 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:39.983372 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:39.983546 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:39.984134 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:39.984142 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:39.984146 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:39.984537 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:39.984699 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:39.985846 (+ 1147us) server_negotiation.cc:299] Negotiation successful
0929 20:29:39.986122 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:29:39.987830 4314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:39.943009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:38887 (local address 127.28.244.129:33657)
0929 20:29:39.967827 (+ 24818us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:39.967852 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:39.967989 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:39.968524 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:39.968528 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:39.969190 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:39.969600 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:39.969613 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.971009 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.971014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:39.972799 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:39.972816 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.973987 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.973994 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:39.974186 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:39.974836 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:39.974858 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:39.978508 (+ 3650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:39.982467 (+ 3959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:39.982477 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:39.982498 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:39.983247 (+ 749us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:39.983694 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:39.983702 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:39.983708 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:39.983992 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:39.984792 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:39.984803 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:39.985377 (+ 574us) client_negotiation.cc:769] Sending connection context
0929 20:29:39.986809 (+ 1432us) client_negotiation.cc:240] Negotiation successful
0929 20:29:39.987094 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24633,"mutex_wait_us":378,"spinlock_wait_cycles":23296,"thread_start_us":24662,"threads_started":1}
I20260929 20:29:39.988281 4315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:39.943008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.28.244.129:35633)
0929 20:29:39.958770 (+ 15762us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:39.958820 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:39.961259 (+ 2439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:39.965268 (+ 4009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:39.965282 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:39.966201 (+ 919us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:39.967128 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:39.967160 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.968974 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.968987 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:39.971843 (+ 2856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:39.971861 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.973163 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.973170 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:39.973470 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:39.974318 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:39.974358 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:39.978146 (+ 3788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:39.982909 (+ 4763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:39.982918 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:39.982936 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:39.983519 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:39.984026 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:39.984030 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:39.984049 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:39.984222 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:39.984957 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:39.984962 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:39.985404 (+ 442us) client_negotiation.cc:769] Sending connection context
0929 20:29:39.985708 (+ 304us) client_negotiation.cc:240] Negotiation successful
0929 20:29:39.986294 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15438,"mutex_wait_us":70,"thread_start_us":27586,"threads_started":1}
I20260929 20:29:39.988582 4317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:39.944604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:35633 (local address 127.28.244.190:39805)
0929 20:29:39.964620 (+ 20016us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:39.964624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:39.964641 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:29:39.964696 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:39.964700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:39.964770 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:39.964909 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:39.967361 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.968698 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.972057 (+ 3359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.972725 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.972847 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:39.978459 (+ 5612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:39.978467 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:39.978472 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:39.978518 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:39.982745 (+ 4227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:39.983679 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:39.983687 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:39.983693 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:39.983871 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:39.984367 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:39.984372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:39.984375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:39.984716 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:39.984867 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:39.987930 (+ 3063us) server_negotiation.cc:299] Negotiation successful
0929 20:29:39.988181 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19912,"thread_start_us":19829,"threads_started":1}
I20260929 20:29:39.989082 4312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:39.942401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:42319 (local address 127.28.244.189:44795)
0929 20:29:39.957776 (+ 15375us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:39.957785 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:39.971217 (+ 13432us) server_negotiation.cc:407] Connection header received
0929 20:29:39.971405 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:39.971410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:39.971513 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:39.971610 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:39.972993 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.974063 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.976117 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.976750 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.976899 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:39.981482 (+ 4583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:39.981490 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:39.981494 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:39.981539 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:39.985122 (+ 3583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:39.985934 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:39.985939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:39.985943 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:39.986098 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:39.986523 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:39.986528 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:39.986530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:39.987165 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:39.987316 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:39.988254 (+ 938us) server_negotiation.cc:299] Negotiation successful
0929 20:29:39.988701 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15174,"thread_start_us":15411,"threads_started":1}
I20260929 20:29:39.989743 4313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:39.943465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:44795 (local address 127.28.244.129:42319)
0929 20:29:39.971075 (+ 27610us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:39.971093 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:39.971232 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:39.971844 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:39.971849 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:39.972456 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:39.972818 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:39.972826 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.974220 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.974224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:39.975910 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:39.975933 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.976933 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.976940 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:39.977134 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:39.977737 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:39.977760 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:39.981212 (+ 3452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:39.985280 (+ 4068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:39.985284 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:39.985287 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:39.985799 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:39.986228 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:39.986234 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:39.986238 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:39.986397 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:39.987375 (+ 978us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:39.987379 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:39.987522 (+ 143us) client_negotiation.cc:769] Sending connection context
0929 20:29:39.987673 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 20:29:39.989418 (+ 1745us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27440,"thread_start_us":16406,"threads_started":1}
I20260929 20:29:39.990182 4309 heartbeater.cc:347] Connected to a master server at 127.28.244.190:39805
I20260929 20:29:39.990226 4308 heartbeater.cc:347] Connected to a master server at 127.28.244.188:38887
I20260929 20:29:39.990552 4309 heartbeater.cc:464] Registering TS with master...
I20260929 20:29:39.990549 4308 heartbeater.cc:464] Registering TS with master...
I20260929 20:29:39.990624 4318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:39.945183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47748 (local address 127.28.244.130:40059)
0929 20:29:39.964325 (+ 19142us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:39.964332 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:39.965878 (+ 1546us) server_negotiation.cc:407] Connection header received
0929 20:29:39.965980 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:39.965989 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:39.966616 (+ 627us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:39.966773 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:39.969043 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.971385 (+ 2342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.973434 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:39.974362 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:39.974526 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:39.980970 (+ 6444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:39.980996 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:39.981019 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:39.981072 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:39.985462 (+ 4390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:39.986472 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:39.986481 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:39.986484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:39.986658 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:39.987107 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:39.987112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:39.987115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:39.987538 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:39.987844 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:39.989872 (+ 2028us) server_negotiation.cc:299] Negotiation successful
0929 20:29:39.990164 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18979,"thread_start_us":18861,"threads_started":1}
I20260929 20:29:39.990784 4310 heartbeater.cc:347] Connected to a master server at 127.28.244.189:44795
I20260929 20:29:39.990890 4310 heartbeater.cc:464] Registering TS with master...
I20260929 20:29:39.991736 4310 heartbeater.cc:511] Master 127.28.244.189:44795 requested a full tablet report, sending...
I20260929 20:29:39.991756 4308 heartbeater.cc:511] Master 127.28.244.188:38887 requested a full tablet report, sending...
I20260929 20:29:39.991885 3659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:29:39.992300 4309 heartbeater.cc:511] Master 127.28.244.190:39805 requested a full tablet report, sending...
I20260929 20:29:39.994154 3940 ts_manager.cc:194] Registered new tserver with Master: e9b791bada31439da862ec8bf5a97488 (127.28.244.129:38387)
I20260929 20:29:39.994563 4127 ts_manager.cc:194] Registered new tserver with Master: e9b791bada31439da862ec8bf5a97488 (127.28.244.129:38387)
I20260929 20:29:39.995800 3340 ts_manager.cc:194] Re-registered known tserver with Master: e9b791bada31439da862ec8bf5a97488 (127.28.244.129:38387)
I20260929 20:29:39.997756 3340 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.244.129@KRBTEST.COM'} at 127.28.244.129:35633
I20260929 20:29:40.080658 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 3536
I20260929 20:29:40.097468 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.130:40059
--local_ip_for_outbound_sockets=127.28.244.130
--tserver_master_addrs=127.28.244.190:39805
--webserver_port=38871
--webserver_interface=127.28.244.130
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.130.keytab
--principal=oryx/127.28.244.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.244.190:39805,127.28.244.189:44795
--tserver_master_addrs=127.28.244.190:39805,127.28.244.189:44795,127.28.244.188:38887 with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:29:40.338624 4085 sys_catalog.cc:455] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c23310ae5174458f8743b50aefe320bc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: false } } peers { permanent_uuid: "7bfc4052f1a84dd184643c71ac194b23" member_type: NON_VOTER last_known_addr { host: "127.28.244.188" port: 38887 } attrs { promote: true } } }
I20260929 20:29:40.339051 4085 sys_catalog.cc:458] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [sys.catalog]: This master's current role is: LEADER
I20260929 20:29:40.340662 4321 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc: attempting to promote NON_VOTER 7bfc4052f1a84dd184643c71ac194b23 to VOTER
I20260929 20:29:40.341256 4321 consensus_queue.cc:237] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: false } } peers { permanent_uuid: "7bfc4052f1a84dd184643c71ac194b23" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 38887 } attrs { promote: false } }
I20260929 20:29:40.343384 4137 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [term 1 LEARNER]: Refusing update from remote peer c23310ae5174458f8743b50aefe320bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 20:29:40.343449 3950 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [term 1 FOLLOWER]: Refusing update from remote peer c23310ae5174458f8743b50aefe320bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 20:29:40.344108 4321 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bfc4052f1a84dd184643c71ac194b23" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 38887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 20:29:40.344539 4322 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "985844716800427a89b43c509229630c" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 20:29:40.347970 4321 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7bfc4052f1a84dd184643c71ac194b23 (127.28.244.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: false } } peers { permanent_uuid: "7bfc4052f1a84dd184643c71ac194b23" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 38887 } attrs { promote: false } } }
I20260929 20:29:40.348768 3950 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7bfc4052f1a84dd184643c71ac194b23 (127.28.244.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: false } } peers { permanent_uuid: "7bfc4052f1a84dd184643c71ac194b23" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 38887 } attrs { promote: false } } }
I20260929 20:29:40.354897 4136 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7bfc4052f1a84dd184643c71ac194b23 (127.28.244.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: false } } peers { permanent_uuid: "7bfc4052f1a84dd184643c71ac194b23" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 38887 } attrs { promote: false } } }
I20260929 20:29:40.356343 4321 sys_catalog.cc:455] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c23310ae5174458f8743b50aefe320bc" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: false } } peers { permanent_uuid: "7bfc4052f1a84dd184643c71ac194b23" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 38887 } attrs { promote: false } } }
I20260929 20:29:40.356518 4085 sys_catalog.cc:455] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c23310ae5174458f8743b50aefe320bc" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: false } } peers { permanent_uuid: "7bfc4052f1a84dd184643c71ac194b23" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 38887 } attrs { promote: false } } }
I20260929 20:29:40.356712 4085 sys_catalog.cc:458] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [sys.catalog]: This master's current role is: LEADER
I20260929 20:29:40.356739 4321 sys_catalog.cc:458] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [sys.catalog]: This master's current role is: LEADER
I20260929 20:29:40.356984 4085 sys_catalog.cc:455] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c23310ae5174458f8743b50aefe320bc" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: false } } peers { permanent_uuid: "7bfc4052f1a84dd184643c71ac194b23" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 38887 } attrs { promote: false } } }
I20260929 20:29:40.357227 4085 sys_catalog.cc:458] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc [sys.catalog]: This master's current role is: LEADER
I20260929 20:29:40.357798 4327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c23310ae5174458f8743b50aefe320bc" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: false } } peers { permanent_uuid: "7bfc4052f1a84dd184643c71ac194b23" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 38887 } attrs { promote: false } } }
I20260929 20:29:40.358206 4327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:29:40.365044 4328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c23310ae5174458f8743b50aefe320bc" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: false } } peers { permanent_uuid: "7bfc4052f1a84dd184643c71ac194b23" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 38887 } attrs { promote: false } } }
I20260929 20:29:40.365546 4328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 20:29:40.468057 4320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:40.468344 4320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:40.468461 4320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:40.475980 4320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:40.476091 4320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:40.476269 4320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.130
I20260929 20:29:40.486692 4320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.130.keytab
--principal=oryx/127.28.244.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.130:40059
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.244.130
--webserver_port=38871
--tserver_master_addrs=127.28.244.190:39805,127.28.244.189:44795,127.28.244.188:38887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.130
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:40.488251 4320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:40.489616 4320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:40.503491 4333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:40.503516 4336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:40.503477 4334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:40.504448 4320 server_base.cc:1034] running on GCE node
I20260929 20:29:40.505080 4320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:40.506039 4320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:40.507239 4320 hybrid_clock.cc:648] HybridClock initialized: now 1790713780507190 us; error 36 us; skew 500 ppm
I20260929 20:29:40.507580 4320 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:40 dist-test-slave-p29l krb5kdc[3296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713780, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.244.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:40.510689 4320 init.cc:377] Logged in from keytab as oryx/127.28.244.130@KRBTEST.COM (short username oryx)
I20260929 20:29:40.512317 4320 webserver.cc:460] Webserver started at http://127.28.244.130:38871/ using document root <none> and password file <none>
I20260929 20:29:40.513027 4320 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:40.513185 4320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:40.518628 4320 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 20:29:40.521911 4343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:40.523540 4320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:29:40.523703 4320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-1/wal
uuid: "5c245511f25e4a97901c08fdf79dfe74"
format_stamp: "Formatted at 2026-09-29 20:29:37 on dist-test-slave-p29l"
server_key: "07a04d4127834047663414dc06ab78a8"
server_key_iv: "664ab6b21b5f46defdb37c015fa58a8d"
server_key_version: "encryptionkey@0"
I20260929 20:29:40.524745 4320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:40.576535 4320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:40.580896 4320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:40.581138 4320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:40.582142 4320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:29:40.583542 4320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:29:40.583629 4320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:40.583711 4320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:29:40.583765 4320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:40.616966 4320 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.130:40059
I20260929 20:29:40.617091 4455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.130:40059 every 8 connection(s)
I20260929 20:29:40.618502 4320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
I20260929 20:29:40.620095 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 4320
I20260929 20:29:40.620465 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 3669
Sep 29 20:29:40 dist-test-slave-p29l krb5kdc[3296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.244.131@KRBTEST.COM
I20260929 20:29:40.652478 4461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:40.622233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38010 (local address 127.28.244.131:36837)
0929 20:29:40.624764 (+ 2531us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:40.624770 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:40.624794 (+ 24us) server_negotiation.cc:407] Connection header received
0929 20:29:40.624900 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:40.624911 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:40.625202 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:40.628167 (+ 2965us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:40.628530 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:40.630665 (+ 2135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:40.633867 (+ 3202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:40.635327 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:40.636377 (+ 1050us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:40.643846 (+ 7469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:40.643866 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:40.643884 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:40.643934 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:40.648041 (+ 4107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:40.649044 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:40.649054 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:40.649057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:40.649219 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:40.649722 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:40.649726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:40.649729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:40.650169 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:40.650475 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:40.651714 (+ 1239us) server_negotiation.cc:299] Negotiation successful
0929 20:29:40.652016 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2371,"thread_start_us":2232,"threads_started":1}
I20260929 20:29:40.653990 3792 generic_service.cc:196] Checking for leaks (request via RPC)
Sep 29 20:29:40 dist-test-slave-p29l krb5kdc[3296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713780, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.244.130@KRBTEST.COM for oryx/127.28.244.189@KRBTEST.COM
Sep 29 20:29:40 dist-test-slave-p29l krb5kdc[3296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713780, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.244.130@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM
Sep 29 20:29:40 dist-test-slave-p29l krb5kdc[3296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713780, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.244.130@KRBTEST.COM for oryx/127.28.244.188@KRBTEST.COM
I20260929 20:29:40.667501 4462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:40.622933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:44613 (local address 127.28.244.189:44795)
0929 20:29:40.624283 (+ 1350us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:40.624293 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:40.642366 (+ 18073us) server_negotiation.cc:407] Connection header received
0929 20:29:40.646511 (+ 4145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:40.646520 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:40.646649 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:40.646788 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:40.650089 (+ 3301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:40.651275 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:40.654257 (+ 2982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:40.654736 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:40.654886 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:40.659526 (+ 4640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:40.659535 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:40.659539 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:40.659590 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:40.663227 (+ 3637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:40.664362 (+ 1135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:40.664371 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:40.664374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:40.664532 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:40.665174 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:40.665180 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:40.665183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:40.665672 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:40.665844 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:40.666698 (+ 854us) server_negotiation.cc:299] Negotiation successful
0929 20:29:40.666938 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1194,"thread_start_us":1073,"threads_started":1}
I20260929 20:29:40.668766 4466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:40.623671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:44795 (local address 127.28.244.130:44613)
0929 20:29:40.642175 (+ 18504us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:40.642243 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:40.645201 (+ 2958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:40.647240 (+ 2039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:40.647255 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:40.648827 (+ 1572us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:40.649848 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:40.649876 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:40.651481 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:40.651485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:40.654119 (+ 2634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:40.654135 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:40.654992 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:40.655002 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:40.655261 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:40.656098 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:40.656137 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:40.659227 (+ 3090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:40.663404 (+ 4177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:40.663415 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:40.663435 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:40.664208 (+ 773us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:40.664697 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:40.664702 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:40.664706 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:40.665007 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:40.665926 (+ 919us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:40.665940 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:40.666293 (+ 353us) client_negotiation.cc:769] Sending connection context
0929 20:29:40.666594 (+ 301us) client_negotiation.cc:240] Negotiation successful
0929 20:29:40.667155 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18101,"mutex_wait_us":393,"spinlock_wait_cycles":20352,"thread_start_us":21814,"threads_started":1}
I20260929 20:29:40.668926 4463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:40.623871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:34493 (local address 127.28.244.190:39805)
0929 20:29:40.647323 (+ 23452us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:40.647331 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:40.647352 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:29:40.647417 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:40.647421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:40.647501 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:40.647630 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:40.650101 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:40.651270 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:40.654347 (+ 3077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:40.655343 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:40.655500 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:40.660432 (+ 4932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:40.660437 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:40.660440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:40.660524 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:40.665213 (+ 4689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:40.666013 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:40.666019 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:40.666023 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:40.666172 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:40.666596 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:40.666601 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:40.666605 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:40.666921 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:40.667092 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:40.668303 (+ 1211us) server_negotiation.cc:299] Negotiation successful
0929 20:29:40.668548 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23300,"thread_start_us":23884,"threads_started":1}
I20260929 20:29:40.669360 4464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:40.623671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.28.244.130:34493)
0929 20:29:40.645984 (+ 22313us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:40.646009 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:40.646162 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:40.647659 (+ 1497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:40.647663 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:40.648693 (+ 1030us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:40.649846 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:40.649875 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:40.651469 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:40.651474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:40.654134 (+ 2660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:40.654152 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:40.656026 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:40.656033 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:40.656238 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:40.656976 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:40.657000 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:40.660243 (+ 3243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:40.665332 (+ 5089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:40.665340 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:40.665346 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:40.665861 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:40.666299 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:40.666304 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:40.666308 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:40.666473 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:40.667152 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:40.667157 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:40.667309 (+ 152us) client_negotiation.cc:769] Sending connection context
0929 20:29:40.667452 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 20:29:40.667750 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22105,"mutex_wait_us":501,"thread_start_us":23920,"threads_started":1}
I20260929 20:29:40.670890 4457 heartbeater.cc:347] Connected to a master server at 127.28.244.190:39805
I20260929 20:29:40.671245 4457 heartbeater.cc:464] Registering TS with master...
I20260929 20:29:40.671938 4458 heartbeater.cc:347] Connected to a master server at 127.28.244.189:44795
I20260929 20:29:40.672276 4458 heartbeater.cc:464] Registering TS with master...
I20260929 20:29:40.672322 4457 heartbeater.cc:511] Master 127.28.244.190:39805 requested a full tablet report, sending...
I20260929 20:29:40.673076 4458 heartbeater.cc:511] Master 127.28.244.189:44795 requested a full tablet report, sending...
I20260929 20:29:40.674454 3340 ts_manager.cc:194] Re-registered known tserver with Master: 5c245511f25e4a97901c08fdf79dfe74 (127.28.244.130:40059)
I20260929 20:29:40.674544 3940 ts_manager.cc:194] Registered new tserver with Master: 5c245511f25e4a97901c08fdf79dfe74 (127.28.244.130:40059)
I20260929 20:29:40.675772 3340 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.244.130@KRBTEST.COM'} at 127.28.244.130:34493
I20260929 20:29:40.676645 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:40.624490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:38887 (local address 127.28.244.130:51245)
0929 20:29:40.648393 (+ 23903us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:40.648412 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:40.648534 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:40.649103 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:40.649107 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:40.649877 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:40.650206 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:40.650217 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:40.651741 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:40.651746 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:40.660033 (+ 8287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:40.660056 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:40.662136 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:40.662144 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:40.662380 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:40.663454 (+ 1074us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:40.663484 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:40.668491 (+ 5007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:40.673572 (+ 5081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:40.673577 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:40.673580 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:40.674066 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:40.674603 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:40.674609 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:40.674613 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:40.674787 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:40.675518 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:40.675534 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:40.675709 (+ 175us) client_negotiation.cc:769] Sending connection context
0929 20:29:40.675874 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 20:29:40.676178 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23741,"mutex_wait_us":6095,"thread_start_us":17163,"threads_started":1}
I20260929 20:29:40.676674 4467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:40.625010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:51245 (local address 127.28.244.188:38887)
0929 20:29:40.646759 (+ 21749us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:40.646767 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:40.648508 (+ 1741us) server_negotiation.cc:407] Connection header received
0929 20:29:40.648640 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:40.648648 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:40.648739 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:40.648858 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:40.650446 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:40.651571 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:40.660818 (+ 9247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:40.661642 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:40.661761 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:40.668739 (+ 6978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:40.668748 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:40.668754 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:40.668818 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:40.673392 (+ 4574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:40.674220 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:40.674225 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:40.674228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:40.674442 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:40.674928 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:40.674933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:40.674936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:40.675308 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:40.675460 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:40.676079 (+ 619us) server_negotiation.cc:299] Negotiation successful
0929 20:29:40.676299 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21619,"thread_start_us":21775,"threads_started":1}
I20260929 20:29:40.677784 4456 heartbeater.cc:347] Connected to a master server at 127.28.244.188:38887
I20260929 20:29:40.677959 4456 heartbeater.cc:464] Registering TS with master...
I20260929 20:29:40.678288 4456 heartbeater.cc:511] Master 127.28.244.188:38887 requested a full tablet report, sending...
I20260929 20:29:40.679342 4127 ts_manager.cc:194] Registered new tserver with Master: 5c245511f25e4a97901c08fdf79dfe74 (127.28.244.130:40059)
I20260929 20:29:40.750388 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 3669
I20260929 20:29:40.767408 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.131:36837
--local_ip_for_outbound_sockets=127.28.244.131
--tserver_master_addrs=127.28.244.190:39805
--webserver_port=45271
--webserver_interface=127.28.244.131
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.131.keytab
--principal=oryx/127.28.244.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.244.190:39805,127.28.244.189:44795
--tserver_master_addrs=127.28.244.190:39805,127.28.244.189:44795,127.28.244.188:38887 with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:29:41.000886 4309 heartbeater.cc:503] Master 127.28.244.190:39805 was elected leader, sending a full tablet report...
W20260929 20:29:41.107849 4469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:41.108204 4469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:41.108381 4469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:41.115718 4469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:41.115919 4469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:41.116102 4469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.131
I20260929 20:29:41.126417 4469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.131.keytab
--principal=oryx/127.28.244.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.131:36837
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.244.131
--webserver_port=45271
--tserver_master_addrs=127.28.244.190:39805,127.28.244.189:44795,127.28.244.188:38887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.131
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:41.128047 4469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:41.129559 4469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:41.144050 4475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:41.144050 4474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:41.144510 4477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:41.145332 4469 server_base.cc:1034] running on GCE node
I20260929 20:29:41.146049 4469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:41.147094 4469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:41.148332 4469 hybrid_clock.cc:648] HybridClock initialized: now 1790713781148257 us; error 70 us; skew 500 ppm
I20260929 20:29:41.148687 4469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:41 dist-test-slave-p29l krb5kdc[3296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713781, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.244.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:41.152206 4469 init.cc:377] Logged in from keytab as oryx/127.28.244.131@KRBTEST.COM (short username oryx)
I20260929 20:29:41.153879 4469 webserver.cc:460] Webserver started at http://127.28.244.131:45271/ using document root <none> and password file <none>
I20260929 20:29:41.154603 4469 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:41.154717 4469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:41.160444 4469 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 20:29:41.163125 4484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:41.164582 4469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:29:41.164749 4469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-2/wal
uuid: "7a7f8f2d242848f1af37b960ac84c810"
format_stamp: "Formatted at 2026-09-29 20:29:37 on dist-test-slave-p29l"
server_key: "3d9b19663d4490f50e852c7846fdbfdd"
server_key_iv: "8596d006e7470b62ab9b75e0403140bc"
server_key_version: "encryptionkey@0"
I20260929 20:29:41.165827 4469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:41.187749 4469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:41.192806 4469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:41.193128 4469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:41.194226 4469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:29:41.195701 4469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:29:41.195794 4469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:41.195957 4469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:29:41.196043 4469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:41.231181 4469 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.131:36837
I20260929 20:29:41.231272 4596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.131:36837 every 8 connection(s)
I20260929 20:29:41.232707 4469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
I20260929 20:29:41.239513 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 4469
Sep 29 20:29:41 dist-test-slave-p29l krb5kdc[3296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713781, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.244.131@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM
Sep 29 20:29:41 dist-test-slave-p29l krb5kdc[3296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713781, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.244.131@KRBTEST.COM for oryx/127.28.244.188@KRBTEST.COM
Sep 29 20:29:41 dist-test-slave-p29l krb5kdc[3296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713781, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.244.131@KRBTEST.COM for oryx/127.28.244.189@KRBTEST.COM
I20260929 20:29:41.270089 4601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:41.235682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:43061 (local address 127.28.244.190:39805)
0929 20:29:41.236177 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:41.236187 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:41.237848 (+ 1661us) server_negotiation.cc:407] Connection header received
0929 20:29:41.241186 (+ 3338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:41.241196 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:41.241314 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:41.241422 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:41.244285 (+ 2863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.245646 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.248865 (+ 3219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.249734 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.249854 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:41.260392 (+ 10538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:41.260400 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:41.260405 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:41.260448 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:41.264189 (+ 3741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:41.265574 (+ 1385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:41.265582 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:41.265587 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:41.266014 (+ 427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:41.266747 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:41.266753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:41.266758 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:41.267155 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:41.267301 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:41.268764 (+ 1463us) server_negotiation.cc:299] Negotiation successful
0929 20:29:41.269329 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":400,"threads_started":1}
I20260929 20:29:41.272955 4603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:41.236432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.28.244.131:43061)
0929 20:29:41.237647 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:41.237717 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:41.240814 (+ 3097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:41.242054 (+ 1240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:41.242070 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:41.243010 (+ 940us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:41.244024 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:41.244061 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.245907 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.245915 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:41.248652 (+ 2737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:41.248674 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.250614 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.250626 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:41.251116 (+ 490us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:41.256529 (+ 5413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:41.256567 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:41.260093 (+ 3526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:41.264345 (+ 4252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:41.264355 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:41.264375 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:41.265411 (+ 1036us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:41.266175 (+ 764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:41.266179 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:41.266183 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:41.266598 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:41.267499 (+ 901us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:41.267508 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:41.268076 (+ 568us) client_negotiation.cc:769] Sending connection context
0929 20:29:41.270716 (+ 2640us) client_negotiation.cc:240] Negotiation successful
0929 20:29:41.271388 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"spinlock_wait_cycles":8960,"thread_start_us":3540,"threads_started":1}
I20260929 20:29:41.276078 4606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:41.237963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:39861 (local address 127.28.244.188:38887)
0929 20:29:41.256362 (+ 18399us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:41.256372 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:41.256824 (+ 452us) server_negotiation.cc:407] Connection header received
0929 20:29:41.256894 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:41.256900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:41.257011 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:41.257115 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:41.259455 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.260531 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.263214 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.263914 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.264068 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:41.268309 (+ 4241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:41.268318 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:41.268325 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:41.268371 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:41.272821 (+ 4450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:41.274067 (+ 1246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:41.274073 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:41.274077 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:41.274224 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:41.274599 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:41.274605 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:41.274608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:41.274900 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:41.275049 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:41.275455 (+ 406us) server_negotiation.cc:299] Negotiation successful
0929 20:29:41.275662 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18167,"thread_start_us":18039,"threads_started":1}
I20260929 20:29:41.276117 4605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:41.237378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:38887 (local address 127.28.244.131:39861)
0929 20:29:41.256536 (+ 19158us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:41.256557 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:41.256703 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:41.257291 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:41.257296 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:41.258941 (+ 1645us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:41.259264 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:41.259276 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.260706 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.260712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:41.263020 (+ 2308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:41.263041 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.264340 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.264346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:41.264566 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:41.265177 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:41.265204 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:41.268073 (+ 2869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:41.272995 (+ 4922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:41.272999 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:41.273003 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:41.273930 (+ 927us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:41.274348 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:41.274353 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:41.274357 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:41.274488 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:41.275031 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:41.275036 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:41.275187 (+ 151us) client_negotiation.cc:769] Sending connection context
0929 20:29:41.275359 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 20:29:41.275658 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19026,"mutex_wait_us":127,"thread_start_us":18736,"threads_started":1}
I20260929 20:29:41.277618 4597 heartbeater.cc:347] Connected to a master server at 127.28.244.188:38887
I20260929 20:29:41.277958 4597 heartbeater.cc:464] Registering TS with master...
I20260929 20:29:41.278264 4602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:41.235992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:52129 (local address 127.28.244.189:44795)
0929 20:29:41.257059 (+ 21067us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:41.257068 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:41.257094 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:29:41.257173 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:41.257181 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:41.257286 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:41.257867 (+ 581us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:41.258833 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.260421 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.263598 (+ 3177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.264641 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.264773 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:41.269377 (+ 4604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:41.269387 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:41.269394 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:41.269448 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:41.274485 (+ 5037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:41.275496 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:41.275504 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:41.275510 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:41.275686 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:41.276234 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:41.276241 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:41.276247 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:41.276671 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:41.276822 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:41.277436 (+ 614us) server_negotiation.cc:299] Negotiation successful
0929 20:29:41.277721 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20901,"thread_start_us":20793,"threads_started":1}
I20260929 20:29:41.278887 4597 heartbeater.cc:511] Master 127.28.244.188:38887 requested a full tablet report, sending...
I20260929 20:29:41.279647 4604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:41.236432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:44795 (local address 127.28.244.131:52129)
0929 20:29:41.251247 (+ 14815us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:41.251271 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:41.251428 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:41.257552 (+ 6124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:41.257574 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:41.258319 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:41.258640 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:41.258654 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.260650 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.260658 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:41.263378 (+ 2720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:41.263399 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.265192 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.265199 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:41.265399 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:41.265995 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:41.266019 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:41.269081 (+ 3062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:41.274672 (+ 5591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:41.274680 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:41.274686 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:41.275322 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:41.275873 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:41.275881 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:41.275888 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:41.276091 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:41.276927 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:41.276935 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:41.277123 (+ 188us) client_negotiation.cc:769] Sending connection context
0929 20:29:41.278586 (+ 1463us) client_negotiation.cc:240] Negotiation successful
0929 20:29:41.279016 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14613,"mutex_wait_us":430,"spinlock_wait_cycles":62720,"thread_start_us":13944,"threads_started":1}
I20260929 20:29:41.284374 4599 heartbeater.cc:347] Connected to a master server at 127.28.244.189:44795
I20260929 20:29:41.284514 4599 heartbeater.cc:464] Registering TS with master...
I20260929 20:29:41.284536 4127 ts_manager.cc:194] Registered new tserver with Master: 7a7f8f2d242848f1af37b960ac84c810 (127.28.244.131:36837)
I20260929 20:29:41.285024 4599 heartbeater.cc:511] Master 127.28.244.189:44795 requested a full tablet report, sending...
I20260929 20:29:41.286300 3940 ts_manager.cc:194] Registered new tserver with Master: 7a7f8f2d242848f1af37b960ac84c810 (127.28.244.131:36837)
I20260929 20:29:41.289825 4598 heartbeater.cc:347] Connected to a master server at 127.28.244.190:39805
I20260929 20:29:41.289983 4598 heartbeater.cc:464] Registering TS with master...
I20260929 20:29:41.290511 4598 heartbeater.cc:511] Master 127.28.244.190:39805 requested a full tablet report, sending...
I20260929 20:29:41.292062 3340 ts_manager.cc:194] Re-registered known tserver with Master: 7a7f8f2d242848f1af37b960ac84c810 (127.28.244.131:36837)
I20260929 20:29:41.293706 3340 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.244.131@KRBTEST.COM'} at 127.28.244.131:43061
I20260929 20:29:41.307720 4601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:41.267426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33410 (local address 127.28.244.190:39805)
0929 20:29:41.270617 (+ 3191us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:41.270627 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:41.288762 (+ 18135us) server_negotiation.cc:407] Connection header received
0929 20:29:41.289038 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:41.289045 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:41.289151 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:41.289219 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:41.292163 (+ 2944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.293378 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.295179 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.295845 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.295947 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:41.300709 (+ 4762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:41.300715 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:41.300719 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:41.300764 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:41.304448 (+ 3684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:41.305309 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:41.305317 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:41.305325 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:41.305455 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:41.305894 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:41.305900 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:41.305903 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:41.306315 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:41.306600 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:41.306997 (+ 397us) server_negotiation.cc:299] Negotiation successful
0929 20:29:41.307274 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3060,"thread_start_us":20658,"threads_started":1}
I20260929 20:29:41.311628 4602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:41.289045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34290 (local address 127.28.244.189:44795)
0929 20:29:41.289270 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:41.289279 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:41.295377 (+ 6098us) server_negotiation.cc:407] Connection header received
0929 20:29:41.295607 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:41.295612 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:41.295703 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:41.295773 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:41.298264 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.299489 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.301743 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.302187 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.302332 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:41.305259 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:41.305268 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:41.305275 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:41.305318 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:41.308408 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:41.309173 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:41.309179 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:41.309182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:41.309331 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:41.309784 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:41.309789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:41.309793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:41.310144 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:41.310425 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:41.311002 (+ 577us) server_negotiation.cc:299] Negotiation successful
0929 20:29:41.311253 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 20:29:41.313242 4606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:41.285987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49866 (local address 127.28.244.188:38887)
0929 20:29:41.286163 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:41.286168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:41.295152 (+ 8984us) server_negotiation.cc:407] Connection header received
0929 20:29:41.295430 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:41.295435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:41.295543 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:41.295617 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:41.298480 (+ 2863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.299367 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.301719 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.302423 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.302567 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:41.305902 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:41.305910 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:41.305916 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:41.305961 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:41.309295 (+ 3334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:41.310064 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:41.310069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:41.310073 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:41.310222 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:41.310656 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:41.310661 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:41.310665 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:41.311037 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:41.311292 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:41.312088 (+ 796us) server_negotiation.cc:299] Negotiation successful
0929 20:29:41.312297 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:29:41.313215 3340 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33410:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:29:41.320371 3340 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:29:41.350401 4623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:41.342090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:40059 (local address 127.0.0.1:47756)
0929 20:29:41.342728 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:41.342751 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:41.342894 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:41.344146 (+ 1252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:41.344153 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:41.344189 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:29:41.344686 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:41.344697 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.346268 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.346276 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:41.347689 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:41.347702 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.349820 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.349825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:41.349959 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:41.349985 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 20:29:41.350054 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 20:29:41.350158 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":635,"threads_started":1}
I20260929 20:29:41.351110 4624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:41.342562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47756 (local address 127.28.244.130:40059)
0929 20:29:41.343121 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:41.343132 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:41.343165 (+ 33us) server_negotiation.cc:407] Connection header received
0929 20:29:41.343284 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:41.343296 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:41.343605 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:41.343867 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:29:41.344876 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.346124 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.347902 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.349109 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.349269 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:41.349489 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:41.350745 (+ 1256us) server_negotiation.cc:299] Negotiation successful
0929 20:29:41.350911 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":660,"threads_started":1}
I20260929 20:29:41.355379 4391 tablet_service.cc:1467] Processing CreateTablet for tablet 29a44ae03b0b4fa5aefc6af8b7cdabb9 (DEFAULT_TABLE table=test-table [id=43ee8d9d974a41f2a1a91cbba77392c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:29:41.356889 4391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29a44ae03b0b4fa5aefc6af8b7cdabb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:41.366889 4627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:41.343902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:36837 (local address 127.0.0.1:38020)
0929 20:29:41.344522 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:41.344541 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:41.344672 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:41.357521 (+ 12849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:41.357530 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:41.357577 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:29:41.358002 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:41.358019 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.362331 (+ 4312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.362339 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:41.363900 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:41.363916 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.366010 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.366016 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:41.366185 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:41.366227 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 20:29:41.366364 (+ 137us) client_negotiation.cc:240] Negotiation successful
0929 20:29:41.366507 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":780,"threads_started":1}
I20260929 20:29:41.366880 4628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:41.344511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38020 (local address 127.28.244.131:36837)
0929 20:29:41.356393 (+ 11882us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:41.356408 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:41.356458 (+ 50us) server_negotiation.cc:407] Connection header received
0929 20:29:41.356594 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:41.356607 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:41.356950 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:41.358296 (+ 1346us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:29:41.358680 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.360920 (+ 2240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.364143 (+ 3223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.365729 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.365989 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:41.366168 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:41.366427 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 20:29:41.366577 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11488,"thread_start_us":12642,"threads_started":1}
I20260929 20:29:41.369941 4625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:41.343075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:38387 (local address 127.0.0.1:34940)
0929 20:29:41.343700 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:41.343721 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:41.343867 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:41.362010 (+ 18143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:41.362015 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:41.362044 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:29:41.362563 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:41.362581 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.364576 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.364582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:41.366732 (+ 2150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:41.366749 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.369148 (+ 2399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.369155 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:41.369322 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:41.369359 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 20:29:41.369436 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 20:29:41.369587 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":424,"threads_started":1}
I20260929 20:29:41.371138 4626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:41.343351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34940 (local address 127.28.244.129:38387)
0929 20:29:41.361530 (+ 18179us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:41.361537 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:41.361558 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:29:41.361636 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:41.361641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:41.361816 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:41.362017 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:29:41.362774 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.364404 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.367314 (+ 2910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.368914 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.370350 (+ 1436us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:41.370539 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:41.370610 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 20:29:41.370793 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17988,"thread_start_us":17878,"threads_started":1}
I20260929 20:29:41.371932 4629 tablet_bootstrap.cc:492] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 5c245511f25e4a97901c08fdf79dfe74: Bootstrap starting.
I20260929 20:29:41.370808 4532 tablet_service.cc:1467] Processing CreateTablet for tablet 29a44ae03b0b4fa5aefc6af8b7cdabb9 (DEFAULT_TABLE table=test-table [id=43ee8d9d974a41f2a1a91cbba77392c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:29:41.373366 4532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29a44ae03b0b4fa5aefc6af8b7cdabb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:41.374965 4243 tablet_service.cc:1467] Processing CreateTablet for tablet 29a44ae03b0b4fa5aefc6af8b7cdabb9 (DEFAULT_TABLE table=test-table [id=43ee8d9d974a41f2a1a91cbba77392c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:29:41.375777 4629 tablet_bootstrap.cc:654] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 5c245511f25e4a97901c08fdf79dfe74: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:41.376207 4243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29a44ae03b0b4fa5aefc6af8b7cdabb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:41.377110 4629 log.cc:824] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 5c245511f25e4a97901c08fdf79dfe74: Log is configured to *not* fsync() on all Append() calls
I20260929 20:29:41.380326 4629 tablet_bootstrap.cc:492] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 5c245511f25e4a97901c08fdf79dfe74: No bootstrap required, opened a new log
I20260929 20:29:41.380628 4629 ts_tablet_manager.cc:1397] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 5c245511f25e4a97901c08fdf79dfe74: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 20:29:41.388234 4631 tablet_bootstrap.cc:492] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P e9b791bada31439da862ec8bf5a97488: Bootstrap starting.
I20260929 20:29:41.389710 4632 tablet_bootstrap.cc:492] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 7a7f8f2d242848f1af37b960ac84c810: Bootstrap starting.
I20260929 20:29:41.391656 4629 raft_consensus.cc:348] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 5c245511f25e4a97901c08fdf79dfe74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c245511f25e4a97901c08fdf79dfe74" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40059 } } peers { permanent_uuid: "e9b791bada31439da862ec8bf5a97488" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38387 } } peers { permanent_uuid: "7a7f8f2d242848f1af37b960ac84c810" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36837 } }
I20260929 20:29:41.392182 4629 raft_consensus.cc:374] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 5c245511f25e4a97901c08fdf79dfe74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:41.392295 4629 raft_consensus.cc:729] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 5c245511f25e4a97901c08fdf79dfe74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c245511f25e4a97901c08fdf79dfe74, State: Initialized, Role: FOLLOWER
I20260929 20:29:41.393173 4631 tablet_bootstrap.cc:654] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P e9b791bada31439da862ec8bf5a97488: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:41.393020 4629 consensus_queue.cc:260] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 5c245511f25e4a97901c08fdf79dfe74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c245511f25e4a97901c08fdf79dfe74" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40059 } } peers { permanent_uuid: "e9b791bada31439da862ec8bf5a97488" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38387 } } peers { permanent_uuid: "7a7f8f2d242848f1af37b960ac84c810" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36837 } }
I20260929 20:29:41.393453 4632 tablet_bootstrap.cc:654] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 7a7f8f2d242848f1af37b960ac84c810: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:41.394594 4632 log.cc:824] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 7a7f8f2d242848f1af37b960ac84c810: Log is configured to *not* fsync() on all Append() calls
I20260929 20:29:41.394594 4631 log.cc:824] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P e9b791bada31439da862ec8bf5a97488: Log is configured to *not* fsync() on all Append() calls
I20260929 20:29:41.394865 4629 ts_tablet_manager.cc:1428] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 5c245511f25e4a97901c08fdf79dfe74: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260929 20:29:41.395130 4457 heartbeater.cc:503] Master 127.28.244.190:39805 was elected leader, sending a full tablet report...
I20260929 20:29:41.403996 4632 tablet_bootstrap.cc:492] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 7a7f8f2d242848f1af37b960ac84c810: No bootstrap required, opened a new log
I20260929 20:29:41.404095 4631 tablet_bootstrap.cc:492] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P e9b791bada31439da862ec8bf5a97488: No bootstrap required, opened a new log
I20260929 20:29:41.404317 4631 ts_tablet_manager.cc:1397] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P e9b791bada31439da862ec8bf5a97488: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.003s
I20260929 20:29:41.404316 4632 ts_tablet_manager.cc:1397] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 7a7f8f2d242848f1af37b960ac84c810: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.012s
I20260929 20:29:41.415431 4632 raft_consensus.cc:348] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 7a7f8f2d242848f1af37b960ac84c810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c245511f25e4a97901c08fdf79dfe74" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40059 } } peers { permanent_uuid: "e9b791bada31439da862ec8bf5a97488" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38387 } } peers { permanent_uuid: "7a7f8f2d242848f1af37b960ac84c810" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36837 } }
I20260929 20:29:41.415431 4631 raft_consensus.cc:348] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P e9b791bada31439da862ec8bf5a97488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c245511f25e4a97901c08fdf79dfe74" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40059 } } peers { permanent_uuid: "e9b791bada31439da862ec8bf5a97488" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38387 } } peers { permanent_uuid: "7a7f8f2d242848f1af37b960ac84c810" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36837 } }
I20260929 20:29:41.415977 4632 raft_consensus.cc:374] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 7a7f8f2d242848f1af37b960ac84c810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:41.416092 4632 raft_consensus.cc:729] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 7a7f8f2d242848f1af37b960ac84c810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a7f8f2d242848f1af37b960ac84c810, State: Initialized, Role: FOLLOWER
I20260929 20:29:41.416311 4631 raft_consensus.cc:374] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P e9b791bada31439da862ec8bf5a97488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:41.416432 4631 raft_consensus.cc:729] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P e9b791bada31439da862ec8bf5a97488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9b791bada31439da862ec8bf5a97488, State: Initialized, Role: FOLLOWER
I20260929 20:29:41.416824 4632 consensus_queue.cc:260] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 7a7f8f2d242848f1af37b960ac84c810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c245511f25e4a97901c08fdf79dfe74" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40059 } } peers { permanent_uuid: "e9b791bada31439da862ec8bf5a97488" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38387 } } peers { permanent_uuid: "7a7f8f2d242848f1af37b960ac84c810" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36837 } }
I20260929 20:29:41.417001 4631 consensus_queue.cc:260] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P e9b791bada31439da862ec8bf5a97488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c245511f25e4a97901c08fdf79dfe74" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40059 } } peers { permanent_uuid: "e9b791bada31439da862ec8bf5a97488" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38387 } } peers { permanent_uuid: "7a7f8f2d242848f1af37b960ac84c810" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36837 } }
I20260929 20:29:41.418902 4632 ts_tablet_manager.cc:1428] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 7a7f8f2d242848f1af37b960ac84c810: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260929 20:29:41.418913 4631 ts_tablet_manager.cc:1428] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P e9b791bada31439da862ec8bf5a97488: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260929 20:29:41.419195 4598 heartbeater.cc:503] Master 127.28.244.190:39805 was elected leader, sending a full tablet report...
W20260929 20:29:41.443255 4311 tablet.cc:2369] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P e9b791bada31439da862ec8bf5a97488: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:29:41.485752 4600 tablet.cc:2369] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 7a7f8f2d242848f1af37b960ac84c810: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:29:41.529115 4636 raft_consensus.cc:482] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 7a7f8f2d242848f1af37b960ac84c810 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:29:41.529450 4636 raft_consensus.cc:504] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 7a7f8f2d242848f1af37b960ac84c810 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c245511f25e4a97901c08fdf79dfe74" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40059 } } peers { permanent_uuid: "e9b791bada31439da862ec8bf5a97488" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38387 } } peers { permanent_uuid: "7a7f8f2d242848f1af37b960ac84c810" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36837 } }
I20260929 20:29:41.531267 4636 leader_election.cc:288] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 7a7f8f2d242848f1af37b960ac84c810 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c245511f25e4a97901c08fdf79dfe74 (127.28.244.130:40059), e9b791bada31439da862ec8bf5a97488 (127.28.244.129:38387)
I20260929 20:29:41.538251 4624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:41.531623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:38995 (local address 127.28.244.130:40059)
0929 20:29:41.531879 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:41.531887 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:41.531987 (+ 100us) server_negotiation.cc:407] Connection header received
0929 20:29:41.532287 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:41.532292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:41.532385 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:41.532527 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:29:41.532907 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.533730 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.535932 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.537053 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.537247 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:41.537325 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:41.537822 (+ 497us) server_negotiation.cc:299] Negotiation successful
0929 20:29:41.537961 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 20:29:41.538237 4626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:41.532353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:58607 (local address 127.28.244.129:38387)
0929 20:29:41.532577 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:41.532585 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:41.532602 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:29:41.532672 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:41.532675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:41.532730 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:41.532829 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:29:41.533217 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.534164 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.536199 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.537231 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.537414 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:41.537499 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:41.537828 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 20:29:41.537967 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:29:41.538296 4605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:41.532107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:38387 (local address 127.28.244.131:58607)
0929 20:29:41.532248 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:41.532271 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:41.532451 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:41.532845 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:41.532849 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:41.532873 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:29:41.533060 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:41.533070 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.534302 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.534307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:41.536039 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:41.536051 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.537423 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.537429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:41.537654 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:41.537693 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 20:29:41.537792 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 20:29:41.537993 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":55}
I20260929 20:29:41.538975 4604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:41.531643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:40059 (local address 127.28.244.131:38995)
0929 20:29:41.531876 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:41.531899 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:41.532071 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:41.532545 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:41.532549 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:41.532576 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:29:41.532763 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:41.532772 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.533894 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.533900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:41.535746 (+ 1846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:41.535759 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.537247 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.537253 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:41.537626 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:41.537689 (+ 63us) client_negotiation.cc:769] Sending connection context
0929 20:29:41.537793 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 20:29:41.538329 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":62}
I20260929 20:29:41.539875 4263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29a44ae03b0b4fa5aefc6af8b7cdabb9" candidate_uuid: "7a7f8f2d242848f1af37b960ac84c810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9b791bada31439da862ec8bf5a97488" is_pre_election: true
I20260929 20:29:41.539986 4411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29a44ae03b0b4fa5aefc6af8b7cdabb9" candidate_uuid: "7a7f8f2d242848f1af37b960ac84c810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c245511f25e4a97901c08fdf79dfe74" is_pre_election: true
I20260929 20:29:41.540462 4263 raft_consensus.cc:2454] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P e9b791bada31439da862ec8bf5a97488 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a7f8f2d242848f1af37b960ac84c810 in term 0.
I20260929 20:29:41.540537 4411 raft_consensus.cc:2454] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 5c245511f25e4a97901c08fdf79dfe74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a7f8f2d242848f1af37b960ac84c810 in term 0.
I20260929 20:29:41.541267 4487 leader_election.cc:302] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 7a7f8f2d242848f1af37b960ac84c810 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a7f8f2d242848f1af37b960ac84c810, e9b791bada31439da862ec8bf5a97488; no voters:
I20260929 20:29:41.541623 4636 raft_consensus.cc:2781] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 7a7f8f2d242848f1af37b960ac84c810 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:29:41.541791 4636 raft_consensus.cc:482] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 7a7f8f2d242848f1af37b960ac84c810 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:29:41.541903 4636 raft_consensus.cc:3037] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 7a7f8f2d242848f1af37b960ac84c810 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:41.544742 4636 raft_consensus.cc:504] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 7a7f8f2d242848f1af37b960ac84c810 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c245511f25e4a97901c08fdf79dfe74" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40059 } } peers { permanent_uuid: "e9b791bada31439da862ec8bf5a97488" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38387 } } peers { permanent_uuid: "7a7f8f2d242848f1af37b960ac84c810" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36837 } }
I20260929 20:29:41.545405 4636 leader_election.cc:288] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 7a7f8f2d242848f1af37b960ac84c810 [CANDIDATE]: Term 1 election: Requested vote from peers 5c245511f25e4a97901c08fdf79dfe74 (127.28.244.130:40059), e9b791bada31439da862ec8bf5a97488 (127.28.244.129:38387)
I20260929 20:29:41.545918 4411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29a44ae03b0b4fa5aefc6af8b7cdabb9" candidate_uuid: "7a7f8f2d242848f1af37b960ac84c810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c245511f25e4a97901c08fdf79dfe74"
I20260929 20:29:41.545993 4263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29a44ae03b0b4fa5aefc6af8b7cdabb9" candidate_uuid: "7a7f8f2d242848f1af37b960ac84c810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9b791bada31439da862ec8bf5a97488"
I20260929 20:29:41.546139 4411 raft_consensus.cc:3037] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 5c245511f25e4a97901c08fdf79dfe74 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:41.546252 4263 raft_consensus.cc:3037] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P e9b791bada31439da862ec8bf5a97488 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:41.548676 4411 raft_consensus.cc:2454] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 5c245511f25e4a97901c08fdf79dfe74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a7f8f2d242848f1af37b960ac84c810 in term 1.
I20260929 20:29:41.548858 4263 raft_consensus.cc:2454] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P e9b791bada31439da862ec8bf5a97488 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a7f8f2d242848f1af37b960ac84c810 in term 1.
I20260929 20:29:41.549110 4487 leader_election.cc:302] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 7a7f8f2d242848f1af37b960ac84c810 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c245511f25e4a97901c08fdf79dfe74, 7a7f8f2d242848f1af37b960ac84c810; no voters:
I20260929 20:29:41.549405 4636 raft_consensus.cc:2781] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 7a7f8f2d242848f1af37b960ac84c810 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:29:41.549862 4636 raft_consensus.cc:686] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 7a7f8f2d242848f1af37b960ac84c810 [term 1 LEADER]: Becoming Leader. State: Replica: 7a7f8f2d242848f1af37b960ac84c810, State: Running, Role: LEADER
I20260929 20:29:41.550482 4636 consensus_queue.cc:237] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 7a7f8f2d242848f1af37b960ac84c810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c245511f25e4a97901c08fdf79dfe74" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40059 } } peers { permanent_uuid: "e9b791bada31439da862ec8bf5a97488" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38387 } } peers { permanent_uuid: "7a7f8f2d242848f1af37b960ac84c810" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36837 } }
I20260929 20:29:41.557210 3339 catalog_manager.cc:5495] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 7a7f8f2d242848f1af37b960ac84c810 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a7f8f2d242848f1af37b960ac84c810 (127.28.244.131). New cstate: current_term: 1 leader_uuid: "7a7f8f2d242848f1af37b960ac84c810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c245511f25e4a97901c08fdf79dfe74" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9b791bada31439da862ec8bf5a97488" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a7f8f2d242848f1af37b960ac84c810" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260929 20:29:41.595026 4628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:41.589118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38032 (local address 127.28.244.131:36837)
0929 20:29:41.589303 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:41.589308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:41.589387 (+ 79us) server_negotiation.cc:407] Connection header received
0929 20:29:41.589656 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:41.589661 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:41.589762 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:41.589953 (+ 191us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:29:41.590373 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.591275 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.592850 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.593279 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.593388 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:41.593929 (+ 541us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:29:41.594499 (+ 570us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:29:41.594620 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:41.594731 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 20:29:41.594807 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 20:29:41.604061 4263 raft_consensus.cc:1261] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P e9b791bada31439da862ec8bf5a97488 [term 1 FOLLOWER]: Refusing update from remote peer 7a7f8f2d242848f1af37b960ac84c810: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:29:41.604144 4411 raft_consensus.cc:1261] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 5c245511f25e4a97901c08fdf79dfe74 [term 1 FOLLOWER]: Refusing update from remote peer 7a7f8f2d242848f1af37b960ac84c810: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:29:41.605141 4638 consensus_queue.cc:1035] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 7a7f8f2d242848f1af37b960ac84c810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c245511f25e4a97901c08fdf79dfe74" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:29:41.605623 4636 consensus_queue.cc:1035] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 7a7f8f2d242848f1af37b960ac84c810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9b791bada31439da862ec8bf5a97488" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:29:41.623351 4642 mvcc.cc:205] Tried to move back new op lower bound from 7334763649438765056 to 7334763649235517440. Current Snapshot: MvccSnapshot[applied={T|T < 7334763649438765056}]
I20260929 20:29:41.649417 3340 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:29:41.649775 3340 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:29:41.657505 3340 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c23310ae5174458f8743b50aefe320bc: Sending DeleteTablet for 3 replicas of tablet 29a44ae03b0b4fa5aefc6af8b7cdabb9
I20260929 20:29:41.659044 4532 tablet_service.cc:1514] Processing DeleteTablet for tablet 29a44ae03b0b4fa5aefc6af8b7cdabb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:29:41 UTC) from {username='oryx', principal='oryx/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:38020
I20260929 20:29:41.659348 4391 tablet_service.cc:1514] Processing DeleteTablet for tablet 29a44ae03b0b4fa5aefc6af8b7cdabb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:29:41 UTC) from {username='oryx', principal='oryx/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:47756
I20260929 20:29:41.659989 4243 tablet_service.cc:1514] Processing DeleteTablet for tablet 29a44ae03b0b4fa5aefc6af8b7cdabb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:29:41 UTC) from {username='oryx', principal='oryx/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:34940
I20260929 20:29:41.661633 4653 tablet_replica.cc:330] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P e9b791bada31439da862ec8bf5a97488: stopping tablet replica
I20260929 20:29:41.662175 4653 raft_consensus.cc:2229] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P e9b791bada31439da862ec8bf5a97488 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:29:41.663039 4653 raft_consensus.cc:2258] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P e9b791bada31439da862ec8bf5a97488 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:29:41.663633 4652 tablet_replica.cc:330] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 5c245511f25e4a97901c08fdf79dfe74: stopping tablet replica
I20260929 20:29:41.664529 4652 raft_consensus.cc:2229] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 5c245511f25e4a97901c08fdf79dfe74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:29:41.665210 4652 raft_consensus.cc:2258] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 5c245511f25e4a97901c08fdf79dfe74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:29:41.666191 4653 ts_tablet_manager.cc:1905] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P e9b791bada31439da862ec8bf5a97488: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:29:41.674239 4652 ts_tablet_manager.cc:1905] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 5c245511f25e4a97901c08fdf79dfe74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:29:41.675171 4653 ts_tablet_manager.cc:1918] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P e9b791bada31439da862ec8bf5a97488: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:29:41.675364 4653 log.cc:1196] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P e9b791bada31439da862ec8bf5a97488: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-0/wal/wals/29a44ae03b0b4fa5aefc6af8b7cdabb9
I20260929 20:29:41.675997 4653 ts_tablet_manager.cc:1939] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P e9b791bada31439da862ec8bf5a97488: Deleting consensus metadata
I20260929 20:29:41.678246 3327 catalog_manager.cc:4841] TS e9b791bada31439da862ec8bf5a97488 (127.28.244.129:38387): tablet 29a44ae03b0b4fa5aefc6af8b7cdabb9 (table test-table [id=43ee8d9d974a41f2a1a91cbba77392c9]) successfully deleted
I20260929 20:29:41.679468 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 3309
I20260929 20:29:41.680859 3371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:29:41.683528 4651 tablet_replica.cc:330] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 7a7f8f2d242848f1af37b960ac84c810: stopping tablet replica
I20260929 20:29:41.686442 4651 raft_consensus.cc:2229] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 7a7f8f2d242848f1af37b960ac84c810 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:29:41.686708 4652 ts_tablet_manager.cc:1918] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 5c245511f25e4a97901c08fdf79dfe74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:29:41.686932 4652 log.cc:1196] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 5c245511f25e4a97901c08fdf79dfe74: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-1/wal/wals/29a44ae03b0b4fa5aefc6af8b7cdabb9
I20260929 20:29:41.687594 4652 ts_tablet_manager.cc:1939] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 5c245511f25e4a97901c08fdf79dfe74: Deleting consensus metadata
I20260929 20:29:41.689718 4651 raft_consensus.cc:2258] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 7a7f8f2d242848f1af37b960ac84c810 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:29:41.693480 4651 ts_tablet_manager.cc:1905] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 7a7f8f2d242848f1af37b960ac84c810: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:29:41.701678 4651 ts_tablet_manager.cc:1918] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 7a7f8f2d242848f1af37b960ac84c810: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:29:41.701917 4651 log.cc:1196] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 7a7f8f2d242848f1af37b960ac84c810: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-2/wal/wals/29a44ae03b0b4fa5aefc6af8b7cdabb9
I20260929 20:29:41.702553 4651 ts_tablet_manager.cc:1939] T 29a44ae03b0b4fa5aefc6af8b7cdabb9 P 7a7f8f2d242848f1af37b960ac84c810: Deleting consensus metadata
I20260929 20:29:41.709081 4602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:41.661231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34296 (local address 127.28.244.189:44795)
0929 20:29:41.661414 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:41.661421 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:41.677976 (+ 16555us) server_negotiation.cc:407] Connection header received
0929 20:29:41.678668 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:41.678674 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:41.678772 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:41.678891 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:41.684045 (+ 5154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.685723 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.690214 (+ 4491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.690889 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.691008 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:41.700561 (+ 9553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:41.700573 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:41.700581 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:41.700643 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:41.705412 (+ 4769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:41.706493 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:41.706499 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:41.706502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:41.706718 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:41.707218 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:41.707225 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:41.707229 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:41.707703 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:41.708031 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:41.708416 (+ 385us) server_negotiation.cc:299] Negotiation successful
0929 20:29:41.708666 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:29:41.726599 4606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:41.678691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49874 (local address 127.28.244.188:38887)
0929 20:29:41.678867 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:41.678873 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:41.709208 (+ 30335us) server_negotiation.cc:407] Connection header received
0929 20:29:41.709470 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:41.709476 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:41.709566 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:41.709638 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:41.711583 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.712984 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.714915 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.715646 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.715791 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:41.719313 (+ 3522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:41.719321 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:41.719327 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:41.719375 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:41.723207 (+ 3832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:41.724125 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:41.724131 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:41.724136 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:41.724297 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:41.724704 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:41.724709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:41.724713 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:41.725135 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:41.725399 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:41.725775 (+ 376us) server_negotiation.cc:299] Negotiation successful
0929 20:29:41.726037 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:29:41.824687 3327 catalog_manager.cc:4841] TS 7a7f8f2d242848f1af37b960ac84c810 (127.28.244.131:36837): tablet 29a44ae03b0b4fa5aefc6af8b7cdabb9 (table test-table [id=43ee8d9d974a41f2a1a91cbba77392c9]) successfully deleted
I20260929 20:29:41.825028 3327 catalog_manager.cc:4841] TS 5c245511f25e4a97901c08fdf79dfe74 (127.28.244.130:40059): tablet 29a44ae03b0b4fa5aefc6af8b7cdabb9 (table test-table [id=43ee8d9d974a41f2a1a91cbba77392c9]) successfully deleted
I20260929 20:29:41.825532 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 3309
I20260929 20:29:41.881597 4606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:41.852949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49876 (local address 127.28.244.188:38887)
0929 20:29:41.853494 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:41.853503 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:41.864086 (+ 10583us) server_negotiation.cc:407] Connection header received
0929 20:29:41.864440 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:41.864446 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:41.864531 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:41.864605 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:41.867311 (+ 2706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.868440 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.870386 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.871097 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.871228 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:41.874983 (+ 3755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:41.874989 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:41.874993 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:41.875039 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:41.878395 (+ 3356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:41.879207 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:41.879214 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:41.879217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:41.879414 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:41.879906 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:41.879914 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:41.879920 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:41.880340 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:41.880591 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:41.880988 (+ 397us) server_negotiation.cc:299] Negotiation successful
0929 20:29:41.881230 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334}
I20260929 20:29:41.882593 4602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:41.853205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34304 (local address 127.28.244.189:44795)
0929 20:29:41.853704 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:41.853713 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:41.863997 (+ 10284us) server_negotiation.cc:407] Connection header received
0929 20:29:41.864442 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:41.864448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:41.864531 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:41.864605 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:41.867329 (+ 2724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.868239 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.870427 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:41.870907 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:41.871074 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:41.874547 (+ 3473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:41.874553 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:41.874556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:41.874593 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:41.878300 (+ 3707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:41.879381 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:41.879387 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:41.879390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:41.879552 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:41.880189 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:41.880195 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:41.880198 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:41.880582 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:41.880770 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:41.881618 (+ 848us) server_negotiation.cc:299] Negotiation successful
0929 20:29:41.882281 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320}
I20260929 20:29:42.636811 4663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:42.635947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.28.244.130:48591)
0929 20:29:42.636413 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:42.636526 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":251,"threads_started":1}
W20260929 20:29:42.637363 4457 heartbeater.cc:644] Failed to heartbeat to 127.28.244.190:39805 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
I20260929 20:29:42.641990 4664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:42.640634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.28.244.129:58569)
0929 20:29:42.641639 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:42.641781 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":872,"thread_start_us":1031,"threads_started":1}
W20260929 20:29:42.642407 4309 heartbeater.cc:644] Failed to heartbeat to 127.28.244.190:39805 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
I20260929 20:29:42.642807 4665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:42.641554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.28.244.131:47449)
0929 20:29:42.642462 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:42.642647 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":813,"thread_start_us":990,"threads_started":1}
W20260929 20:29:42.643132 4598 heartbeater.cc:644] Failed to heartbeat to 127.28.244.190:39805 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
I20260929 20:29:43.158109 4666 raft_consensus.cc:482] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c23310ae5174458f8743b50aefe320bc)
I20260929 20:29:43.158408 4666 raft_consensus.cc:504] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: false } } peers { permanent_uuid: "7bfc4052f1a84dd184643c71ac194b23" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 38887 } attrs { promote: false } }
I20260929 20:29:43.159003 4667 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c23310ae5174458f8743b50aefe320bc)
I20260929 20:29:43.159253 4667 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: false } } peers { permanent_uuid: "7bfc4052f1a84dd184643c71ac194b23" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 38887 } attrs { promote: false } }
I20260929 20:29:43.160017 4666 leader_election.cc:288] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c23310ae5174458f8743b50aefe320bc (127.28.244.190:39805), 7bfc4052f1a84dd184643c71ac194b23 (127.28.244.188:38887)
I20260929 20:29:43.161468 4668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:43.160142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.0.0.1:33810)
0929 20:29:43.160871 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:43.161125 (+ 254us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":592,"threads_started":1}
W20260929 20:29:43.161809 3924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
I20260929 20:29:43.165578 4667 leader_election.cc:288] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c23310ae5174458f8743b50aefe320bc (127.28.244.190:39805), 985844716800427a89b43c509229630c (127.28.244.189:44795)
I20260929 20:29:43.171276 4671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:43.160811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.0.0.1:33814)
0929 20:29:43.169849 (+ 9038us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:43.170067 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8901,"thread_start_us":8775,"threads_started":1}
W20260929 20:29:43.171612 4111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
I20260929 20:29:43.177696 4668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:43.176857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.0.0.1:33828)
0929 20:29:43.177259 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:43.177418 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209}
W20260929 20:29:43.178212 3924 leader_election.cc:334] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c23310ae5174458f8743b50aefe320bc (127.28.244.190:39805): Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
I20260929 20:29:43.180229 4672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:43.179263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.0.0.1:33834)
0929 20:29:43.179511 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:43.179912 (+ 401us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260929 20:29:43.180699 4111 leader_election.cc:334] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c23310ae5174458f8743b50aefe320bc (127.28.244.190:39805): Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
I20260929 20:29:43.185672 4669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:43.160214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:38887 (local address 127.0.0.1:49892)
0929 20:29:43.161621 (+ 1407us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:43.161645 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:43.161818 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:43.177901 (+ 16083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:43.177907 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:43.177938 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:29:43.178389 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:43.178410 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:43.180418 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:43.180424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:43.182538 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:43.182553 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:43.184533 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:43.184542 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:43.184759 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:43.184884 (+ 125us) client_negotiation.cc:769] Sending connection context
0929 20:29:43.184973 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 20:29:43.185176 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1238,"mutex_wait_us":28,"thread_start_us":1066,"threads_started":1}
I20260929 20:29:43.185724 4671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:43.161001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:44795 (local address 127.0.0.1:34318)
0929 20:29:43.171547 (+ 10546us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:43.171571 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:43.171727 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:43.179182 (+ 7455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:43.179187 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:43.179213 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:29:43.179569 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:43.179581 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:43.181467 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:43.181473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:43.183243 (+ 1770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:43.183255 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:43.184694 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:43.184700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:43.184930 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:43.184987 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 20:29:43.185075 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 20:29:43.185200 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10420,"mutex_wait_us":27,"thread_start_us":13228,"threads_started":1}
I20260929 20:29:43.186118 4673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:43.161576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34318 (local address 127.28.244.189:44795)
0929 20:29:43.178842 (+ 17266us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:43.178848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:43.178867 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:29:43.178927 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:43.178931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:43.179019 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:43.179722 (+ 703us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:29:43.179989 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:43.181305 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:43.183401 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:43.184502 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:43.184702 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:43.185568 (+ 866us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:43.185631 (+ 63us) server_negotiation.cc:299] Negotiation successful
0929 20:29:43.185774 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":741,"server-negotiator.queue_time_us":17150,"thread_start_us":16987,"threads_started":1}
I20260929 20:29:43.186702 4670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:43.160581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49892 (local address 127.28.244.188:38887)
0929 20:29:43.177491 (+ 16910us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:43.177503 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:43.177526 (+ 23us) server_negotiation.cc:407] Connection header received
0929 20:29:43.177601 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:43.177609 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:43.177700 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:43.178214 (+ 514us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:29:43.178770 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:43.180253 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:43.182709 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:43.184343 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:43.184494 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:43.184575 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:43.186055 (+ 1480us) server_negotiation.cc:299] Negotiation successful
0929 20:29:43.186271 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":16747,"thread_start_us":16604,"threads_started":1}
I20260929 20:29:43.187290 4136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "985844716800427a89b43c509229630c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7bfc4052f1a84dd184643c71ac194b23" is_pre_election: true
I20260929 20:29:43.187502 3950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7bfc4052f1a84dd184643c71ac194b23" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "985844716800427a89b43c509229630c" is_pre_election: true
I20260929 20:29:43.187698 4136 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 985844716800427a89b43c509229630c in term 1.
I20260929 20:29:43.187878 3950 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bfc4052f1a84dd184643c71ac194b23 in term 1.
I20260929 20:29:43.188241 3925 leader_election.cc:302] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7bfc4052f1a84dd184643c71ac194b23, 985844716800427a89b43c509229630c; no voters: c23310ae5174458f8743b50aefe320bc
I20260929 20:29:43.188656 4666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 20:29:43.188778 4666 raft_consensus.cc:482] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [term 1 FOLLOWER]: Starting leader election (detected failure of leader c23310ae5174458f8743b50aefe320bc)
I20260929 20:29:43.188881 4666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [term 1 FOLLOWER]: Advancing to term 2
I20260929 20:29:43.189142 4114 leader_election.cc:302] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7bfc4052f1a84dd184643c71ac194b23, 985844716800427a89b43c509229630c; no voters: c23310ae5174458f8743b50aefe320bc
I20260929 20:29:43.189718 4667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 20:29:43.189832 4667 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c23310ae5174458f8743b50aefe320bc)
I20260929 20:29:43.189921 4667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [term 1 FOLLOWER]: Advancing to term 2
I20260929 20:29:43.192128 4666 raft_consensus.cc:504] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: false } } peers { permanent_uuid: "7bfc4052f1a84dd184643c71ac194b23" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 38887 } attrs { promote: false } }
I20260929 20:29:43.192958 4666 leader_election.cc:288] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [CANDIDATE]: Term 2 election: Requested vote from peers c23310ae5174458f8743b50aefe320bc (127.28.244.190:39805), 7bfc4052f1a84dd184643c71ac194b23 (127.28.244.188:38887)
I20260929 20:29:43.193122 4667 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: false } } peers { permanent_uuid: "7bfc4052f1a84dd184643c71ac194b23" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 38887 } attrs { promote: false } }
I20260929 20:29:43.193625 4669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:43.193156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.0.0.1:33840)
0929 20:29:43.193349 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:43.193458 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 20:29:43.193817 4667 leader_election.cc:288] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [CANDIDATE]: Term 2 election: Requested vote from peers c23310ae5174458f8743b50aefe320bc (127.28.244.190:39805), 985844716800427a89b43c509229630c (127.28.244.189:44795)
I20260929 20:29:43.194633 4136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "985844716800427a89b43c509229630c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7bfc4052f1a84dd184643c71ac194b23"
I20260929 20:29:43.194824 4669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:43.194335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.0.0.1:33844)
0929 20:29:43.194554 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:43.194648 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260929 20:29:43.194909 4136 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 985844716800427a89b43c509229630c in current term 2: Already voted for candidate 7bfc4052f1a84dd184643c71ac194b23 in this term.
W20260929 20:29:43.195144 3924 leader_election.cc:334] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c23310ae5174458f8743b50aefe320bc (127.28.244.190:39805): Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
I20260929 20:29:43.195295 3950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7bfc4052f1a84dd184643c71ac194b23" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "985844716800427a89b43c509229630c"
I20260929 20:29:43.195385 3925 leader_election.cc:302] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 985844716800427a89b43c509229630c; no voters: 7bfc4052f1a84dd184643c71ac194b23, c23310ae5174458f8743b50aefe320bc
I20260929 20:29:43.195595 3950 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7bfc4052f1a84dd184643c71ac194b23 in current term 2: Already voted for candidate 985844716800427a89b43c509229630c in this term.
I20260929 20:29:43.195778 4666 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 20:29:43.196486 4671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:43.196046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.0.0.1:33860)
0929 20:29:43.196236 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:43.196329 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 20:29:43.197515 4671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:43.197111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.0.0.1:33876)
0929 20:29:43.197297 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:43.197379 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260929 20:29:43.197706 4111 leader_election.cc:334] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c23310ae5174458f8743b50aefe320bc (127.28.244.190:39805): Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
I20260929 20:29:43.197836 4111 leader_election.cc:302] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7bfc4052f1a84dd184643c71ac194b23; no voters: 985844716800427a89b43c509229630c, c23310ae5174458f8743b50aefe320bc
I20260929 20:29:43.198518 4667 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 20:29:43.639536 4676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:43.638243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.28.244.130:45137)
0929 20:29:43.638691 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:43.638780 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":227,"threads_started":1}
I20260929 20:29:43.644119 4677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:43.643175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.28.244.129:56955)
0929 20:29:43.643788 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:43.643937 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":401,"threads_started":1}
I20260929 20:29:43.644547 4678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:43.643691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.28.244.131:36471)
0929 20:29:43.644218 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:43.644338 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":306,"threads_started":1}
I20260929 20:29:44.642103 4679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:44.640930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.28.244.130:50449)
0929 20:29:44.641709 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:44.641821 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":551,"threads_started":1}
I20260929 20:29:44.645681 4680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:44.644965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.28.244.129:56953)
0929 20:29:44.645432 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:44.645530 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":254,"threads_started":1}
I20260929 20:29:44.645995 4681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:44.645289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.28.244.131:51631)
0929 20:29:44.645737 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:44.645836 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":257,"threads_started":1}
I20260929 20:29:44.842759 4682 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:29:44.843127 4682 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: false } } peers { permanent_uuid: "7bfc4052f1a84dd184643c71ac194b23" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 38887 } attrs { promote: false } }
I20260929 20:29:44.843990 4682 leader_election.cc:288] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c23310ae5174458f8743b50aefe320bc (127.28.244.190:39805), 985844716800427a89b43c509229630c (127.28.244.189:44795)
I20260929 20:29:44.844590 3950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7bfc4052f1a84dd184643c71ac194b23" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "985844716800427a89b43c509229630c" is_pre_election: true
I20260929 20:29:44.844815 3950 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bfc4052f1a84dd184643c71ac194b23 in term 2.
I20260929 20:29:44.844969 4683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:44.844042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.0.0.1:34106)
0929 20:29:44.844601 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:44.844691 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":234,"threads_started":1}
I20260929 20:29:44.845254 4114 leader_election.cc:302] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bfc4052f1a84dd184643c71ac194b23, 985844716800427a89b43c509229630c; no voters:
I20260929 20:29:44.845618 4682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 20:29:44.845760 4682 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:29:44.845875 4682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [term 2 FOLLOWER]: Advancing to term 3
I20260929 20:29:44.846556 4683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:44.846149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.0.0.1:34122)
0929 20:29:44.846294 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:44.846434 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 20:29:44.846729 4111 leader_election.cc:334] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c23310ae5174458f8743b50aefe320bc (127.28.244.190:39805): Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
I20260929 20:29:44.848232 4682 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: false } } peers { permanent_uuid: "7bfc4052f1a84dd184643c71ac194b23" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 38887 } attrs { promote: false } }
I20260929 20:29:44.848868 4682 leader_election.cc:288] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [CANDIDATE]: Term 3 election: Requested vote from peers c23310ae5174458f8743b50aefe320bc (127.28.244.190:39805), 985844716800427a89b43c509229630c (127.28.244.189:44795)
I20260929 20:29:44.849434 3950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7bfc4052f1a84dd184643c71ac194b23" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "985844716800427a89b43c509229630c"
I20260929 20:29:44.849584 4683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:44.849073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.0.0.1:34126)
0929 20:29:44.849228 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:44.849407 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 20:29:44.849689 3950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [term 2 FOLLOWER]: Advancing to term 3
I20260929 20:29:44.850618 4683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:44.850233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.0.0.1:34130)
0929 20:29:44.850399 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:44.850483 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 20:29:44.850783 4111 leader_election.cc:334] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer c23310ae5174458f8743b50aefe320bc (127.28.244.190:39805): Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
I20260929 20:29:44.852272 3950 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bfc4052f1a84dd184643c71ac194b23 in term 3.
I20260929 20:29:44.852733 4114 leader_election.cc:302] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7bfc4052f1a84dd184643c71ac194b23, 985844716800427a89b43c509229630c; no voters: c23310ae5174458f8743b50aefe320bc
I20260929 20:29:44.853080 4682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 20:29:44.853585 4682 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [term 3 LEADER]: Becoming Leader. State: Replica: 7bfc4052f1a84dd184643c71ac194b23, State: Running, Role: LEADER
I20260929 20:29:44.854025 4682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: false } } peers { permanent_uuid: "7bfc4052f1a84dd184643c71ac194b23" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 38887 } attrs { promote: false } }
I20260929 20:29:44.856148 4685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bfc4052f1a84dd184643c71ac194b23. Latest consensus state: current_term: 3 leader_uuid: "7bfc4052f1a84dd184643c71ac194b23" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: false } } peers { permanent_uuid: "7bfc4052f1a84dd184643c71ac194b23" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 38887 } attrs { promote: false } } }
I20260929 20:29:44.856523 4685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [sys.catalog]: This master's current role is: LEADER
I20260929 20:29:44.857120 4687 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:29:44.860800 4687 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:29:44.861631 4687 catalog_manager.cc:1259] Loaded cluster ID: bb52180e1176467eb311d00f60c824a6
I20260929 20:29:44.861706 4687 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:29:44.862888 4687 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:29:44.863713 4687 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23: Loaded TSK: 0
I20260929 20:29:44.864713 4687 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:29:44.928263 4126 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:29:44.935737 4126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:29:44.940892 4683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:44.940372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.0.0.1:34154)
0929 20:29:44.940638 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:44.940730 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 20:29:44.940846 3950 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [term 3 FOLLOWER]: Refusing update from remote peer 7bfc4052f1a84dd184643c71ac194b23: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260929 20:29:44.941685 4682 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "985844716800427a89b43c509229630c" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260929 20:29:44.942150 4683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:44.941773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.0.0.1:34160)
0929 20:29:44.941961 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:44.942039 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 20:29:44.942456 4111 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 -> Peer c23310ae5174458f8743b50aefe320bc (127.28.244.190:39805): Couldn't send request to peer c23310ae5174458f8743b50aefe320bc. Status: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 20:29:44.944828 4689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bfc4052f1a84dd184643c71ac194b23. Latest consensus state: current_term: 3 leader_uuid: "7bfc4052f1a84dd184643c71ac194b23" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: false } } peers { permanent_uuid: "7bfc4052f1a84dd184643c71ac194b23" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 38887 } attrs { promote: false } } }
I20260929 20:29:44.945173 4689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:29:44.946326 4685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "7bfc4052f1a84dd184643c71ac194b23" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: false } } peers { permanent_uuid: "7bfc4052f1a84dd184643c71ac194b23" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 38887 } attrs { promote: false } } }
I20260929 20:29:44.946663 4685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23 [sys.catalog]: This master's current role is: LEADER
I20260929 20:29:44.947616 4689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "7bfc4052f1a84dd184643c71ac194b23" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c23310ae5174458f8743b50aefe320bc" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 39805 } } peers { permanent_uuid: "985844716800427a89b43c509229630c" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 44795 } attrs { promote: false } } peers { permanent_uuid: "7bfc4052f1a84dd184643c71ac194b23" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 38887 } attrs { promote: false } } }
I20260929 20:29:44.948021 4689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 985844716800427a89b43c509229630c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:29:44.969167 4697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:44.958134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:40059 (local address 127.0.0.1:47764)
0929 20:29:44.959690 (+ 1556us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:44.959705 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:44.959917 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:44.960534 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:44.960541 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:44.960573 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:29:44.961031 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:44.961047 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:44.962801 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:44.962809 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:44.965604 (+ 2795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:44.965626 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:44.967969 (+ 2343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:44.967977 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:44.968258 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:44.968314 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 20:29:44.968424 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 20:29:44.968627 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1419,"thread_start_us":1283,"threads_started":1}
I20260929 20:29:44.970613 4696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:44.958246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47764 (local address 127.28.244.130:40059)
0929 20:29:44.958808 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:44.958813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:44.960041 (+ 1228us) server_negotiation.cc:407] Connection header received
0929 20:29:44.960133 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:44.960138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:44.960213 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:44.960310 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:29:44.961256 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:44.962599 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:44.965837 (+ 3238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:44.967482 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:44.967693 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:44.967826 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:44.969841 (+ 2015us) server_negotiation.cc:299] Negotiation successful
0929 20:29:44.970149 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":326,"threads_started":1}
I20260929 20:29:44.973104 4391 tablet_service.cc:1467] Processing CreateTablet for tablet 2630fd300b4045afbd67fae88fabf3dd (DEFAULT_TABLE table=test-table [id=8234b4a8b9894d9aa872f91338dc6bf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:29:44.973592 4391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2630fd300b4045afbd67fae88fabf3dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:44.978610 4697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:44.959991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:36837 (local address 127.0.0.1:38040)
0929 20:29:44.969515 (+ 9524us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:44.969539 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:44.969661 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:44.972218 (+ 2557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:44.972223 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:44.972255 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:29:44.972446 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:44.972462 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:44.973777 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:44.973782 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:44.975771 (+ 1989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:44.975786 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:44.977579 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:44.977585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:44.977886 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:44.977928 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 20:29:44.978026 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 20:29:44.978197 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9381,"mutex_wait_us":4,"thread_start_us":12140,"threads_started":1}
I20260929 20:29:44.979372 4683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:44.956482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:38387 (local address 127.0.0.1:34954)
0929 20:29:44.956658 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:44.956678 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:44.956790 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:44.969332 (+ 12542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:44.969341 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:44.969376 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:29:44.969765 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:44.969782 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:44.972450 (+ 2668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:44.972457 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:44.975325 (+ 2868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:44.975347 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:44.978795 (+ 3448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:44.978800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:44.978998 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:44.979022 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 20:29:44.979079 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 20:29:44.979173 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 20:29:44.979539 4699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:44.960428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38040 (local address 127.28.244.131:36837)
0929 20:29:44.971572 (+ 11144us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:44.971581 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:44.971608 (+ 27us) server_negotiation.cc:407] Connection header received
0929 20:29:44.971696 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:44.971705 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:44.971831 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:44.971967 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:29:44.972643 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:44.973599 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:44.975988 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:44.977235 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:44.977391 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:44.977462 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:44.979045 (+ 1583us) server_negotiation.cc:299] Negotiation successful
0929 20:29:44.979202 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10966,"thread_start_us":10831,"threads_started":1}
I20260929 20:29:44.980285 4695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:44.956439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34954 (local address 127.28.244.129:38387)
0929 20:29:44.968667 (+ 12228us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:44.968676 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:44.968705 (+ 29us) server_negotiation.cc:407] Connection header received
0929 20:29:44.968801 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:44.968810 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:44.968932 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:44.969978 (+ 1046us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:29:44.970284 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:44.971790 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:44.975767 (+ 3977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:44.977526 (+ 1759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:44.979645 (+ 2119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:44.979737 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:44.979781 (+ 44us) server_negotiation.cc:299] Negotiation successful
0929 20:29:44.979959 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12024,"thread_start_us":12702,"threads_started":1}
I20260929 20:29:44.980792 4532 tablet_service.cc:1467] Processing CreateTablet for tablet 2630fd300b4045afbd67fae88fabf3dd (DEFAULT_TABLE table=test-table [id=8234b4a8b9894d9aa872f91338dc6bf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:29:44.980983 4243 tablet_service.cc:1467] Processing CreateTablet for tablet 2630fd300b4045afbd67fae88fabf3dd (DEFAULT_TABLE table=test-table [id=8234b4a8b9894d9aa872f91338dc6bf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:29:44.981252 4532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2630fd300b4045afbd67fae88fabf3dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:44.981360 4243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2630fd300b4045afbd67fae88fabf3dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:44.984444 4700 tablet_bootstrap.cc:492] T 2630fd300b4045afbd67fae88fabf3dd P 5c245511f25e4a97901c08fdf79dfe74: Bootstrap starting.
I20260929 20:29:44.985853 4700 tablet_bootstrap.cc:654] T 2630fd300b4045afbd67fae88fabf3dd P 5c245511f25e4a97901c08fdf79dfe74: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:44.987982 4700 tablet_bootstrap.cc:492] T 2630fd300b4045afbd67fae88fabf3dd P 5c245511f25e4a97901c08fdf79dfe74: No bootstrap required, opened a new log
I20260929 20:29:44.988145 4703 tablet_bootstrap.cc:492] T 2630fd300b4045afbd67fae88fabf3dd P e9b791bada31439da862ec8bf5a97488: Bootstrap starting.
I20260929 20:29:44.988222 4700 ts_tablet_manager.cc:1397] T 2630fd300b4045afbd67fae88fabf3dd P 5c245511f25e4a97901c08fdf79dfe74: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 20:29:44.989084 4700 raft_consensus.cc:348] T 2630fd300b4045afbd67fae88fabf3dd P 5c245511f25e4a97901c08fdf79dfe74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b791bada31439da862ec8bf5a97488" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38387 } } peers { permanent_uuid: "5c245511f25e4a97901c08fdf79dfe74" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40059 } } peers { permanent_uuid: "7a7f8f2d242848f1af37b960ac84c810" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36837 } }
I20260929 20:29:44.989374 4700 raft_consensus.cc:374] T 2630fd300b4045afbd67fae88fabf3dd P 5c245511f25e4a97901c08fdf79dfe74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:44.989498 4700 raft_consensus.cc:729] T 2630fd300b4045afbd67fae88fabf3dd P 5c245511f25e4a97901c08fdf79dfe74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c245511f25e4a97901c08fdf79dfe74, State: Initialized, Role: FOLLOWER
I20260929 20:29:44.989768 4700 consensus_queue.cc:260] T 2630fd300b4045afbd67fae88fabf3dd P 5c245511f25e4a97901c08fdf79dfe74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b791bada31439da862ec8bf5a97488" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38387 } } peers { permanent_uuid: "5c245511f25e4a97901c08fdf79dfe74" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40059 } } peers { permanent_uuid: "7a7f8f2d242848f1af37b960ac84c810" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36837 } }
I20260929 20:29:44.990684 4702 tablet_bootstrap.cc:492] T 2630fd300b4045afbd67fae88fabf3dd P 7a7f8f2d242848f1af37b960ac84c810: Bootstrap starting.
I20260929 20:29:44.990710 4700 ts_tablet_manager.cc:1428] T 2630fd300b4045afbd67fae88fabf3dd P 5c245511f25e4a97901c08fdf79dfe74: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 20:29:44.991611 4703 tablet_bootstrap.cc:654] T 2630fd300b4045afbd67fae88fabf3dd P e9b791bada31439da862ec8bf5a97488: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:44.992383 4679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:44.991883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.28.244.130:39891)
0929 20:29:44.992106 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:44.992206 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260929 20:29:44.993160 4702 tablet_bootstrap.cc:654] T 2630fd300b4045afbd67fae88fabf3dd P 7a7f8f2d242848f1af37b960ac84c810: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:44.993757 4679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:44.993309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.28.244.130:41119)
0929 20:29:44.993497 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:44.993598 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 20:29:44.995002 4456 heartbeater.cc:503] Master 127.28.244.188:38887 was elected leader, sending a full tablet report...
I20260929 20:29:44.996304 4702 tablet_bootstrap.cc:492] T 2630fd300b4045afbd67fae88fabf3dd P 7a7f8f2d242848f1af37b960ac84c810: No bootstrap required, opened a new log
I20260929 20:29:44.996469 4702 ts_tablet_manager.cc:1397] T 2630fd300b4045afbd67fae88fabf3dd P 7a7f8f2d242848f1af37b960ac84c810: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260929 20:29:44.997381 4702 raft_consensus.cc:348] T 2630fd300b4045afbd67fae88fabf3dd P 7a7f8f2d242848f1af37b960ac84c810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b791bada31439da862ec8bf5a97488" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38387 } } peers { permanent_uuid: "5c245511f25e4a97901c08fdf79dfe74" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40059 } } peers { permanent_uuid: "7a7f8f2d242848f1af37b960ac84c810" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36837 } }
I20260929 20:29:44.997574 4702 raft_consensus.cc:374] T 2630fd300b4045afbd67fae88fabf3dd P 7a7f8f2d242848f1af37b960ac84c810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:44.997689 4702 raft_consensus.cc:729] T 2630fd300b4045afbd67fae88fabf3dd P 7a7f8f2d242848f1af37b960ac84c810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a7f8f2d242848f1af37b960ac84c810, State: Initialized, Role: FOLLOWER
I20260929 20:29:44.997925 4702 consensus_queue.cc:260] T 2630fd300b4045afbd67fae88fabf3dd P 7a7f8f2d242848f1af37b960ac84c810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b791bada31439da862ec8bf5a97488" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38387 } } peers { permanent_uuid: "5c245511f25e4a97901c08fdf79dfe74" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40059 } } peers { permanent_uuid: "7a7f8f2d242848f1af37b960ac84c810" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36837 } }
I20260929 20:29:44.998680 4702 ts_tablet_manager.cc:1428] T 2630fd300b4045afbd67fae88fabf3dd P 7a7f8f2d242848f1af37b960ac84c810: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 20:29:44.999625 4681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:44.999044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.28.244.131:57423)
0929 20:29:44.999302 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:44.999428 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260929 20:29:45.000347 4703 tablet_bootstrap.cc:492] T 2630fd300b4045afbd67fae88fabf3dd P e9b791bada31439da862ec8bf5a97488: No bootstrap required, opened a new log
I20260929 20:29:45.000573 4703 ts_tablet_manager.cc:1397] T 2630fd300b4045afbd67fae88fabf3dd P e9b791bada31439da862ec8bf5a97488: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.003s
I20260929 20:29:45.000775 4681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:45.000350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.28.244.131:47037)
0929 20:29:45.000525 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:45.000619 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 20:29:45.001417 4597 heartbeater.cc:503] Master 127.28.244.188:38887 was elected leader, sending a full tablet report...
I20260929 20:29:45.001521 4703 raft_consensus.cc:348] T 2630fd300b4045afbd67fae88fabf3dd P e9b791bada31439da862ec8bf5a97488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b791bada31439da862ec8bf5a97488" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38387 } } peers { permanent_uuid: "5c245511f25e4a97901c08fdf79dfe74" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40059 } } peers { permanent_uuid: "7a7f8f2d242848f1af37b960ac84c810" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36837 } }
I20260929 20:29:45.001755 4703 raft_consensus.cc:374] T 2630fd300b4045afbd67fae88fabf3dd P e9b791bada31439da862ec8bf5a97488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:45.001883 4703 raft_consensus.cc:729] T 2630fd300b4045afbd67fae88fabf3dd P e9b791bada31439da862ec8bf5a97488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9b791bada31439da862ec8bf5a97488, State: Initialized, Role: FOLLOWER
I20260929 20:29:45.002182 4703 consensus_queue.cc:260] T 2630fd300b4045afbd67fae88fabf3dd P e9b791bada31439da862ec8bf5a97488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b791bada31439da862ec8bf5a97488" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38387 } } peers { permanent_uuid: "5c245511f25e4a97901c08fdf79dfe74" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40059 } } peers { permanent_uuid: "7a7f8f2d242848f1af37b960ac84c810" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36837 } }
I20260929 20:29:45.002998 4703 ts_tablet_manager.cc:1428] T 2630fd300b4045afbd67fae88fabf3dd P e9b791bada31439da862ec8bf5a97488: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 20:29:45.004045 4680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:45.003529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.28.244.129:46799)
0929 20:29:45.003762 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:45.003885 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260929 20:29:45.004952 4308 heartbeater.cc:503] Master 127.28.244.188:38887 was elected leader, sending a full tablet report...
I20260929 20:29:45.005280 4680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:45.004887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.28.244.129:57241)
0929 20:29:45.005053 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:45.005134 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 20:29:45.126822 4459 tablet.cc:2369] T 2630fd300b4045afbd67fae88fabf3dd P 5c245511f25e4a97901c08fdf79dfe74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:29:45.322831 4704 raft_consensus.cc:482] T 2630fd300b4045afbd67fae88fabf3dd P 5c245511f25e4a97901c08fdf79dfe74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:29:45.323125 4704 raft_consensus.cc:504] T 2630fd300b4045afbd67fae88fabf3dd P 5c245511f25e4a97901c08fdf79dfe74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b791bada31439da862ec8bf5a97488" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38387 } } peers { permanent_uuid: "5c245511f25e4a97901c08fdf79dfe74" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40059 } } peers { permanent_uuid: "7a7f8f2d242848f1af37b960ac84c810" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36837 } }
I20260929 20:29:45.324781 4704 leader_election.cc:288] T 2630fd300b4045afbd67fae88fabf3dd P 5c245511f25e4a97901c08fdf79dfe74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9b791bada31439da862ec8bf5a97488 (127.28.244.129:38387), 7a7f8f2d242848f1af37b960ac84c810 (127.28.244.131:36837)
I20260929 20:29:45.330829 4695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:45.325165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:53885 (local address 127.28.244.129:38387)
0929 20:29:45.325348 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:45.325353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:45.325373 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:29:45.325608 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:45.325614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:45.325693 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:45.325806 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:29:45.326407 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:45.327113 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:45.328991 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:45.329890 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:45.330089 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:45.330168 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:45.330434 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 20:29:45.330564 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 20:29:45.330895 4679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:45.325116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:38387 (local address 127.28.244.130:53885)
0929 20:29:45.325289 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:45.325311 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:45.325448 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:45.325991 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:45.325995 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:45.326022 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:29:45.326261 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:45.326269 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:45.327248 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:45.327255 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:45.328823 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:45.328835 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:45.330098 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:45.330105 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:45.330275 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:45.330299 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 20:29:45.330407 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 20:29:45.330560 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":18}
I20260929 20:29:45.331487 4699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:45.325676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:57251 (local address 127.28.244.131:36837)
0929 20:29:45.325883 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:45.325889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:45.326263 (+ 374us) server_negotiation.cc:407] Connection header received
0929 20:29:45.326495 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:45.326499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:45.326561 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:45.326657 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:29:45.327135 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:45.327865 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:45.329739 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:45.330522 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:45.330702 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:45.330784 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:45.331174 (+ 390us) server_negotiation.cc:299] Negotiation successful
0929 20:29:45.331307 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 20:29:45.331537 4263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2630fd300b4045afbd67fae88fabf3dd" candidate_uuid: "5c245511f25e4a97901c08fdf79dfe74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9b791bada31439da862ec8bf5a97488" is_pre_election: true
I20260929 20:29:45.331837 4263 raft_consensus.cc:2454] T 2630fd300b4045afbd67fae88fabf3dd P e9b791bada31439da862ec8bf5a97488 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c245511f25e4a97901c08fdf79dfe74 in term 0.
I20260929 20:29:45.332223 4709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:45.325480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:36837 (local address 127.28.244.130:57251)
0929 20:29:45.326153 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:45.326167 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:45.326323 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:45.326671 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:45.326675 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:45.326697 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:29:45.326989 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:45.326998 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:45.328000 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:45.328007 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:45.329575 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:45.329587 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:45.330747 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:45.330753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:45.330975 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:45.331025 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 20:29:45.331686 (+ 661us) client_negotiation.cc:240] Negotiation successful
0929 20:29:45.331889 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":1,"thread_start_us":350,"threads_started":1}
I20260929 20:29:45.332428 4346 leader_election.cc:302] T 2630fd300b4045afbd67fae88fabf3dd P 5c245511f25e4a97901c08fdf79dfe74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c245511f25e4a97901c08fdf79dfe74, e9b791bada31439da862ec8bf5a97488; no voters:
I20260929 20:29:45.332835 4704 raft_consensus.cc:2781] T 2630fd300b4045afbd67fae88fabf3dd P 5c245511f25e4a97901c08fdf79dfe74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:29:45.332991 4704 raft_consensus.cc:482] T 2630fd300b4045afbd67fae88fabf3dd P 5c245511f25e4a97901c08fdf79dfe74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:29:45.333097 4704 raft_consensus.cc:3037] T 2630fd300b4045afbd67fae88fabf3dd P 5c245511f25e4a97901c08fdf79dfe74 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:45.333546 4552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2630fd300b4045afbd67fae88fabf3dd" candidate_uuid: "5c245511f25e4a97901c08fdf79dfe74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a7f8f2d242848f1af37b960ac84c810" is_pre_election: true
I20260929 20:29:45.333973 4552 raft_consensus.cc:2454] T 2630fd300b4045afbd67fae88fabf3dd P 7a7f8f2d242848f1af37b960ac84c810 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c245511f25e4a97901c08fdf79dfe74 in term 0.
I20260929 20:29:45.335618 4704 raft_consensus.cc:504] T 2630fd300b4045afbd67fae88fabf3dd P 5c245511f25e4a97901c08fdf79dfe74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b791bada31439da862ec8bf5a97488" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38387 } } peers { permanent_uuid: "5c245511f25e4a97901c08fdf79dfe74" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40059 } } peers { permanent_uuid: "7a7f8f2d242848f1af37b960ac84c810" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36837 } }
I20260929 20:29:45.336228 4704 leader_election.cc:288] T 2630fd300b4045afbd67fae88fabf3dd P 5c245511f25e4a97901c08fdf79dfe74 [CANDIDATE]: Term 1 election: Requested vote from peers e9b791bada31439da862ec8bf5a97488 (127.28.244.129:38387), 7a7f8f2d242848f1af37b960ac84c810 (127.28.244.131:36837)
I20260929 20:29:45.336745 4263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2630fd300b4045afbd67fae88fabf3dd" candidate_uuid: "5c245511f25e4a97901c08fdf79dfe74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9b791bada31439da862ec8bf5a97488"
I20260929 20:29:45.336807 4552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2630fd300b4045afbd67fae88fabf3dd" candidate_uuid: "5c245511f25e4a97901c08fdf79dfe74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a7f8f2d242848f1af37b960ac84c810"
I20260929 20:29:45.336990 4263 raft_consensus.cc:3037] T 2630fd300b4045afbd67fae88fabf3dd P e9b791bada31439da862ec8bf5a97488 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:45.337081 4552 raft_consensus.cc:3037] T 2630fd300b4045afbd67fae88fabf3dd P 7a7f8f2d242848f1af37b960ac84c810 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:45.339046 4263 raft_consensus.cc:2454] T 2630fd300b4045afbd67fae88fabf3dd P e9b791bada31439da862ec8bf5a97488 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c245511f25e4a97901c08fdf79dfe74 in term 1.
I20260929 20:29:45.339316 4552 raft_consensus.cc:2454] T 2630fd300b4045afbd67fae88fabf3dd P 7a7f8f2d242848f1af37b960ac84c810 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c245511f25e4a97901c08fdf79dfe74 in term 1.
I20260929 20:29:45.339496 4346 leader_election.cc:302] T 2630fd300b4045afbd67fae88fabf3dd P 5c245511f25e4a97901c08fdf79dfe74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c245511f25e4a97901c08fdf79dfe74, e9b791bada31439da862ec8bf5a97488; no voters:
I20260929 20:29:45.339890 4704 raft_consensus.cc:2781] T 2630fd300b4045afbd67fae88fabf3dd P 5c245511f25e4a97901c08fdf79dfe74 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:29:45.340348 4704 raft_consensus.cc:686] T 2630fd300b4045afbd67fae88fabf3dd P 5c245511f25e4a97901c08fdf79dfe74 [term 1 LEADER]: Becoming Leader. State: Replica: 5c245511f25e4a97901c08fdf79dfe74, State: Running, Role: LEADER
I20260929 20:29:45.340847 4704 consensus_queue.cc:237] T 2630fd300b4045afbd67fae88fabf3dd P 5c245511f25e4a97901c08fdf79dfe74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b791bada31439da862ec8bf5a97488" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38387 } } peers { permanent_uuid: "5c245511f25e4a97901c08fdf79dfe74" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40059 } } peers { permanent_uuid: "7a7f8f2d242848f1af37b960ac84c810" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36837 } }
I20260929 20:29:45.341491 4709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:45.341019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.28.244.130:40309)
0929 20:29:45.341194 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:45.341318 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 20:29:45.344869 4127 catalog_manager.cc:5495] T 2630fd300b4045afbd67fae88fabf3dd P 5c245511f25e4a97901c08fdf79dfe74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c245511f25e4a97901c08fdf79dfe74 (127.28.244.130). New cstate: current_term: 1 leader_uuid: "5c245511f25e4a97901c08fdf79dfe74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b791bada31439da862ec8bf5a97488" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c245511f25e4a97901c08fdf79dfe74" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a7f8f2d242848f1af37b960ac84c810" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260929 20:29:45.368824 4696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:45.363150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47780 (local address 127.28.244.130:40059)
0929 20:29:45.363356 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:45.363365 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:45.363385 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:29:45.363685 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:45.363690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:45.363772 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:45.364015 (+ 243us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:29:45.364383 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:45.365126 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:45.366470 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:45.366899 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:45.367048 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:45.367467 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:29:45.368211 (+ 744us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:29:45.368362 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:45.368462 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 20:29:45.368546 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 20:29:45.373029 4263 raft_consensus.cc:1261] T 2630fd300b4045afbd67fae88fabf3dd P e9b791bada31439da862ec8bf5a97488 [term 1 FOLLOWER]: Refusing update from remote peer 5c245511f25e4a97901c08fdf79dfe74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:29:45.373284 4552 raft_consensus.cc:1261] T 2630fd300b4045afbd67fae88fabf3dd P 7a7f8f2d242848f1af37b960ac84c810 [term 1 FOLLOWER]: Refusing update from remote peer 5c245511f25e4a97901c08fdf79dfe74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:29:45.373754 4710 consensus_queue.cc:1035] T 2630fd300b4045afbd67fae88fabf3dd P 5c245511f25e4a97901c08fdf79dfe74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9b791bada31439da862ec8bf5a97488" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:29:45.374190 4704 consensus_queue.cc:1035] T 2630fd300b4045afbd67fae88fabf3dd P 5c245511f25e4a97901c08fdf79dfe74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a7f8f2d242848f1af37b960ac84c810" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:29:45.376672 4680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:45.376149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.28.244.129:42047)
0929 20:29:45.376370 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:45.376490 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260929 20:29:45.376813 4681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:45.376210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.28.244.131:55023)
0929 20:29:45.376496 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:45.376617 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260929 20:29:45.389458 4709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:45.388906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.28.244.130:46005)
0929 20:29:45.389144 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:45.389252 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260929 20:29:45.393014 4680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:45.391013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.28.244.129:46207)
0929 20:29:45.392716 (+ 1703us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:45.392820 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1574}
I20260929 20:29:45.393585 4709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:45.391073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.28.244.130:38899)
0929 20:29:45.393313 (+ 2240us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:45.393433 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2134}
I20260929 20:29:45.393769 4715 mvcc.cc:205] Tried to move back new op lower bound from 7334763664881049600 to 7334763664760360960. Current Snapshot: MvccSnapshot[applied={T|T < 7334763664881049600}]
I20260929 20:29:45.395555 4681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:45.395098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.28.244.131:34653)
0929 20:29:45.395286 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:45.395379 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 20:29:45.404950 4126 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:29:45.405256 4126 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:29:45.411206 4126 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7bfc4052f1a84dd184643c71ac194b23: Sending DeleteTablet for 3 replicas of tablet 2630fd300b4045afbd67fae88fabf3dd
I20260929 20:29:45.412487 4243 tablet_service.cc:1514] Processing DeleteTablet for tablet 2630fd300b4045afbd67fae88fabf3dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:29:45 UTC) from {username='oryx', principal='oryx/127.28.244.188@KRBTEST.COM'} at 127.0.0.1:34954
I20260929 20:29:45.412505 4532 tablet_service.cc:1514] Processing DeleteTablet for tablet 2630fd300b4045afbd67fae88fabf3dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:29:45 UTC) from {username='oryx', principal='oryx/127.28.244.188@KRBTEST.COM'} at 127.0.0.1:38040
I20260929 20:29:45.413746 4391 tablet_service.cc:1514] Processing DeleteTablet for tablet 2630fd300b4045afbd67fae88fabf3dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:29:45 UTC) from {username='oryx', principal='oryx/127.28.244.188@KRBTEST.COM'} at 127.0.0.1:47764
I20260929 20:29:45.414263 4724 tablet_replica.cc:330] T 2630fd300b4045afbd67fae88fabf3dd P e9b791bada31439da862ec8bf5a97488: stopping tablet replica
I20260929 20:29:45.414460 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 4164
I20260929 20:29:45.414492 4724 raft_consensus.cc:2229] T 2630fd300b4045afbd67fae88fabf3dd P e9b791bada31439da862ec8bf5a97488 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:29:45.414780 4724 raft_consensus.cc:2258] T 2630fd300b4045afbd67fae88fabf3dd P e9b791bada31439da862ec8bf5a97488 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:29:45.416697 4724 ts_tablet_manager.cc:1905] T 2630fd300b4045afbd67fae88fabf3dd P e9b791bada31439da862ec8bf5a97488: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:29:45.417518 4723 tablet_replica.cc:330] T 2630fd300b4045afbd67fae88fabf3dd P 7a7f8f2d242848f1af37b960ac84c810: stopping tablet replica
I20260929 20:29:45.417732 4723 raft_consensus.cc:2229] T 2630fd300b4045afbd67fae88fabf3dd P 7a7f8f2d242848f1af37b960ac84c810 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:29:45.417783 4683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:45.417119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.0.0.1:34166)
0929 20:29:45.417422 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:45.417557 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260929 20:29:45.418063 4723 raft_consensus.cc:2258] T 2630fd300b4045afbd67fae88fabf3dd P 7a7f8f2d242848f1af37b960ac84c810 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:29:45.419382 4683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:45.418807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.0.0.1:34176)
0929 20:29:45.419068 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:45.419171 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260929 20:29:45.421334 4723 ts_tablet_manager.cc:1905] T 2630fd300b4045afbd67fae88fabf3dd P 7a7f8f2d242848f1af37b960ac84c810: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:29:45.423619 4724 ts_tablet_manager.cc:1918] T 2630fd300b4045afbd67fae88fabf3dd P e9b791bada31439da862ec8bf5a97488: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:29:45.423780 4724 log.cc:1196] T 2630fd300b4045afbd67fae88fabf3dd P e9b791bada31439da862ec8bf5a97488: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-0/wal/wals/2630fd300b4045afbd67fae88fabf3dd
I20260929 20:29:45.424311 4724 ts_tablet_manager.cc:1939] T 2630fd300b4045afbd67fae88fabf3dd P e9b791bada31439da862ec8bf5a97488: Deleting consensus metadata
I20260929 20:29:45.425549 4113 catalog_manager.cc:4841] TS e9b791bada31439da862ec8bf5a97488 (127.28.244.129:38387): tablet 2630fd300b4045afbd67fae88fabf3dd (table test-table [id=8234b4a8b9894d9aa872f91338dc6bf4]) successfully deleted
I20260929 20:29:45.429244 4723 ts_tablet_manager.cc:1918] T 2630fd300b4045afbd67fae88fabf3dd P 7a7f8f2d242848f1af37b960ac84c810: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:29:45.429363 4723 log.cc:1196] T 2630fd300b4045afbd67fae88fabf3dd P 7a7f8f2d242848f1af37b960ac84c810: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-2/wal/wals/2630fd300b4045afbd67fae88fabf3dd
I20260929 20:29:45.429821 4723 ts_tablet_manager.cc:1939] T 2630fd300b4045afbd67fae88fabf3dd P 7a7f8f2d242848f1af37b960ac84c810: Deleting consensus metadata
I20260929 20:29:45.430907 4113 catalog_manager.cc:4841] TS 7a7f8f2d242848f1af37b960ac84c810 (127.28.244.131:36837): tablet 2630fd300b4045afbd67fae88fabf3dd (table test-table [id=8234b4a8b9894d9aa872f91338dc6bf4]) successfully deleted
I20260929 20:29:45.431177 4725 tablet_replica.cc:330] T 2630fd300b4045afbd67fae88fabf3dd P 5c245511f25e4a97901c08fdf79dfe74: stopping tablet replica
I20260929 20:29:45.431399 4725 raft_consensus.cc:2229] T 2630fd300b4045afbd67fae88fabf3dd P 5c245511f25e4a97901c08fdf79dfe74 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:29:45.432097 4725 raft_consensus.cc:2258] T 2630fd300b4045afbd67fae88fabf3dd P 5c245511f25e4a97901c08fdf79dfe74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:29:45.433634 4725 ts_tablet_manager.cc:1905] T 2630fd300b4045afbd67fae88fabf3dd P 5c245511f25e4a97901c08fdf79dfe74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:29:45.441461 4725 ts_tablet_manager.cc:1918] T 2630fd300b4045afbd67fae88fabf3dd P 5c245511f25e4a97901c08fdf79dfe74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:29:45.441677 4725 log.cc:1196] T 2630fd300b4045afbd67fae88fabf3dd P 5c245511f25e4a97901c08fdf79dfe74: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713721201454-29650-0/minicluster-data/ts-1/wal/wals/2630fd300b4045afbd67fae88fabf3dd
I20260929 20:29:45.442158 4725 ts_tablet_manager.cc:1939] T 2630fd300b4045afbd67fae88fabf3dd P 5c245511f25e4a97901c08fdf79dfe74: Deleting consensus metadata
I20260929 20:29:45.443403 4113 catalog_manager.cc:4841] TS 5c245511f25e4a97901c08fdf79dfe74 (127.28.244.130:40059): tablet 2630fd300b4045afbd67fae88fabf3dd (table test-table [id=8234b4a8b9894d9aa872f91338dc6bf4]) successfully deleted
I20260929 20:29:45.444720 4695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:45.415508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34958 (local address 127.28.244.129:38387)
0929 20:29:45.415739 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:45.415749 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:45.420005 (+ 4256us) server_negotiation.cc:407] Connection header received
0929 20:29:45.420328 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:45.420337 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:45.420435 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:45.420522 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:45.423116 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:45.424339 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:45.426314 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:45.426940 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:45.427041 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:45.436498 (+ 9457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:45.436521 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:45.436542 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:45.436592 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:45.440845 (+ 4253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:45.441916 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:45.441927 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:45.441933 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:45.442108 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:45.442612 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:45.442617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:45.442621 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:45.443113 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:45.443432 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:45.444087 (+ 655us) server_negotiation.cc:299] Negotiation successful
0929 20:29:45.444333 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 20:29:45.445526 4303 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:29:45.545526 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 4164
I20260929 20:29:45.564705 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 4320
I20260929 20:29:45.582273 4696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:45.565585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47794 (local address 127.28.244.130:40059)
0929 20:29:45.565769 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:45.565776 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:45.565803 (+ 27us) server_negotiation.cc:407] Connection header received
0929 20:29:45.565928 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:45.565935 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:45.566043 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:45.566162 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:45.567420 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:45.568655 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:45.570938 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:45.572013 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:45.572142 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:45.574799 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:45.574821 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:45.574843 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:45.574889 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:45.579077 (+ 4188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:45.579968 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:45.579978 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:45.579982 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:45.580170 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:45.580574 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:45.580579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:45.580583 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:45.581075 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:45.581392 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:45.581648 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 20:29:45.581880 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 20:29:45.583089 4451 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:29:45.716348 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 4320
I20260929 20:29:45.736938 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 4469
I20260929 20:29:45.750922 4699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:45.737776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38050 (local address 127.28.244.131:36837)
0929 20:29:45.737945 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:45.737950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:45.737969 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:29:45.738147 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:45.738151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:45.738215 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:45.738303 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:45.739878 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:45.740673 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:45.741937 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:45.742347 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:45.742454 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:45.745226 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:45.745243 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:45.745259 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:45.745298 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:45.748433 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:45.749103 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:45.749110 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:45.749112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:45.749234 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:45.749597 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:45.749600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:45.749602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:45.749979 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:45.750233 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:45.750494 (+ 261us) server_negotiation.cc:299] Negotiation successful
0929 20:29:45.750677 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 20:29:45.751664 4592 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:29:45.876338 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 4469
I20260929 20:29:45.877120 4683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:45.876549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.0.0.1:34190)
0929 20:29:45.876823 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:45.876931 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260929 20:29:45.879518 4683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:45.879099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:39805 (local address 127.0.0.1:34198)
0929 20:29:45.879265 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:45.879373 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 20:29:45.900193 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 3802
I20260929 20:29:45.901369 3970 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:29:46.020390 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 3802
I20260929 20:29:46.042410 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 3903
I20260929 20:29:46.043527 4157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:29:46.169023 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 3903
2026-09-29T20:29:46Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11675 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:29:46 dist-test-slave-p29l krb5kdc[4734](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:29:46 dist-test-slave-p29l krb5kdc[4734](info): set up 2 sockets
Sep 29 20:29:46 dist-test-slave-p29l krb5kdc[4734](info): commencing operation
krb5kdc: starting...
W20260929 20:29:47.284463 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:29:47 dist-test-slave-p29l krb5kdc[4734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:29:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:29:47Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.244.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.244.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.244.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.190.keytab.
Entry for principal oryx/127.28.244.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.190.keytab.
I20260929 20:29:47.515581 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:35517
--webserver_interface=127.28.244.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.190.keytab
--principal=oryx/127.28.244.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:34121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:35517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:29:47.850363 4747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:47.850660 4747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:47.850817 4747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:47.858841 4747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:29:47.858949 4747 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 20:29:47.859019 4747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:47.859105 4747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:47.859191 4747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:29:47.859277 4747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:29:47.869375 4747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:34121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:35517
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.190.keytab
--principal=oryx/127.28.244.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:35517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:47.870708 4747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:47.872126 4747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:47.884454 4753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:47.884550 4752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:47.884577 4755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:47.885281 4747 server_base.cc:1034] running on GCE node
I20260929 20:29:47.886118 4747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:47.887703 4747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:47.888971 4747 hybrid_clock.cc:648] HybridClock initialized: now 1790713787888908 us; error 37 us; skew 500 ppm
I20260929 20:29:47.889367 4747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:47 dist-test-slave-p29l krb5kdc[4734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713787, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.244.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:47.892783 4747 init.cc:377] Logged in from keytab as oryx/127.28.244.190@KRBTEST.COM (short username oryx)
I20260929 20:29:47.894361 4747 webserver.cc:460] Webserver started at http://127.28.244.190:36963/ using document root <none> and password file <none>
I20260929 20:29:47.895123 4747 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:47.895252 4747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:47.895579 4747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:47.898764 4747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "535a1a0ea12c439ab74e69d63b55b5ea"
format_stamp: "Formatted at 2026-09-29 20:29:47 on dist-test-slave-p29l"
server_key: "ec11038236b6ae6d6c5ac09353236b68"
server_key_iv: "335fb4bc95a8cd6edca72405a40f17dc"
server_key_version: "encryptionkey@0"
I20260929 20:29:47.899606 4747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "535a1a0ea12c439ab74e69d63b55b5ea"
format_stamp: "Formatted at 2026-09-29 20:29:47 on dist-test-slave-p29l"
server_key: "ec11038236b6ae6d6c5ac09353236b68"
server_key_iv: "335fb4bc95a8cd6edca72405a40f17dc"
server_key_version: "encryptionkey@0"
I20260929 20:29:47.904920 4747 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.003s
I20260929 20:29:47.908748 4762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:47.910236 4747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 20:29:47.910437 4747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "535a1a0ea12c439ab74e69d63b55b5ea"
format_stamp: "Formatted at 2026-09-29 20:29:47 on dist-test-slave-p29l"
server_key: "ec11038236b6ae6d6c5ac09353236b68"
server_key_iv: "335fb4bc95a8cd6edca72405a40f17dc"
server_key_version: "encryptionkey@0"
I20260929 20:29:47.910670 4747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:47.932950 4747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:47.937521 4747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:47.937803 4747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:47.960487 4747 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:35517
I20260929 20:29:47.960575 4823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:35517 every 8 connection(s)
I20260929 20:29:47.961921 4747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:29:47.965814 4824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:47.968365 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 4747
I20260929 20:29:47.968590 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:29:47.969313 29650 external_mini_cluster.cc:1428] Setting key c63b29a81c9c84474670eab979094142
I20260929 20:29:47.978919 4824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 535a1a0ea12c439ab74e69d63b55b5ea: Bootstrap starting.
I20260929 20:29:47.982842 4824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 535a1a0ea12c439ab74e69d63b55b5ea: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:47.984256 4824 log.cc:824] T 00000000000000000000000000000000 P 535a1a0ea12c439ab74e69d63b55b5ea: Log is configured to *not* fsync() on all Append() calls
Sep 29 20:29:47 dist-test-slave-p29l krb5kdc[4734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM
I20260929 20:29:47.987682 4824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 535a1a0ea12c439ab74e69d63b55b5ea: No bootstrap required, opened a new log
I20260929 20:29:47.994894 4827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:47.971701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47554 (local address 127.28.244.190:35517)
0929 20:29:47.972753 (+ 1052us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:47.972766 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:47.972814 (+ 48us) server_negotiation.cc:407] Connection header received
0929 20:29:47.974745 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:47.974775 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:47.975231 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:47.975607 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:47.977424 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:47.978897 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:47.980753 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:47.981457 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:47.981625 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:47.986597 (+ 4972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:47.986606 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:47.986620 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:47.986659 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:47.990232 (+ 3573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:47.991175 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:47.991186 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:47.991197 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:47.991439 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:47.991915 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:47.991919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:47.991921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:47.992550 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:47.992778 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:47.993308 (+ 530us) server_negotiation.cc:299] Negotiation successful
0929 20:29:47.993798 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":780,"thread_start_us":460,"threads_started":1}
I20260929 20:29:47.997290 4824 raft_consensus.cc:348] T 00000000000000000000000000000000 P 535a1a0ea12c439ab74e69d63b55b5ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "535a1a0ea12c439ab74e69d63b55b5ea" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 35517 } }
I20260929 20:29:47.997704 4824 raft_consensus.cc:374] T 00000000000000000000000000000000 P 535a1a0ea12c439ab74e69d63b55b5ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:47.997830 4824 raft_consensus.cc:729] T 00000000000000000000000000000000 P 535a1a0ea12c439ab74e69d63b55b5ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 535a1a0ea12c439ab74e69d63b55b5ea, State: Initialized, Role: FOLLOWER
I20260929 20:29:47.998512 4824 consensus_queue.cc:260] T 00000000000000000000000000000000 P 535a1a0ea12c439ab74e69d63b55b5ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "535a1a0ea12c439ab74e69d63b55b5ea" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 35517 } }
I20260929 20:29:47.998782 4824 raft_consensus.cc:388] T 00000000000000000000000000000000 P 535a1a0ea12c439ab74e69d63b55b5ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:29:47.998919 4824 raft_consensus.cc:482] T 00000000000000000000000000000000 P 535a1a0ea12c439ab74e69d63b55b5ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:29:47.999094 4824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 535a1a0ea12c439ab74e69d63b55b5ea [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:48.001565 4824 raft_consensus.cc:504] T 00000000000000000000000000000000 P 535a1a0ea12c439ab74e69d63b55b5ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "535a1a0ea12c439ab74e69d63b55b5ea" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 35517 } }
I20260929 20:29:48.002100 4824 leader_election.cc:302] T 00000000000000000000000000000000 P 535a1a0ea12c439ab74e69d63b55b5ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 535a1a0ea12c439ab74e69d63b55b5ea; no voters:
I20260929 20:29:48.002678 4824 leader_election.cc:288] T 00000000000000000000000000000000 P 535a1a0ea12c439ab74e69d63b55b5ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:29:48.002873 4829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 535a1a0ea12c439ab74e69d63b55b5ea [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:29:48.004082 4829 raft_consensus.cc:686] T 00000000000000000000000000000000 P 535a1a0ea12c439ab74e69d63b55b5ea [term 1 LEADER]: Becoming Leader. State: Replica: 535a1a0ea12c439ab74e69d63b55b5ea, State: Running, Role: LEADER
I20260929 20:29:48.004607 4829 consensus_queue.cc:237] T 00000000000000000000000000000000 P 535a1a0ea12c439ab74e69d63b55b5ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "535a1a0ea12c439ab74e69d63b55b5ea" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 35517 } }
I20260929 20:29:48.005067 4824 sys_catalog.cc:564] T 00000000000000000000000000000000 P 535a1a0ea12c439ab74e69d63b55b5ea [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:29:48.007777 4831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 535a1a0ea12c439ab74e69d63b55b5ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 535a1a0ea12c439ab74e69d63b55b5ea. Latest consensus state: current_term: 1 leader_uuid: "535a1a0ea12c439ab74e69d63b55b5ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "535a1a0ea12c439ab74e69d63b55b5ea" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 35517 } } }
I20260929 20:29:48.007699 4830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 535a1a0ea12c439ab74e69d63b55b5ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "535a1a0ea12c439ab74e69d63b55b5ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "535a1a0ea12c439ab74e69d63b55b5ea" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 35517 } } }
I20260929 20:29:48.008188 4831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 535a1a0ea12c439ab74e69d63b55b5ea [sys.catalog]: This master's current role is: LEADER
I20260929 20:29:48.008204 4830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 535a1a0ea12c439ab74e69d63b55b5ea [sys.catalog]: This master's current role is: LEADER
I20260929 20:29:48.014802 4838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:29:48.020032 4838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:29:48.029342 4838 catalog_manager.cc:1347] Generated new cluster ID: 5a7ff25e32fd4309ab245e837d243cea
I20260929 20:29:48.029538 4838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:29:48.038615 4838 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:29:48.040498 4838 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:29:48.061679 4838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 535a1a0ea12c439ab74e69d63b55b5ea: Generated new TSK 0
I20260929 20:29:48.062629 4838 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.244.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.244.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.244.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.129.keytab.
Entry for principal oryx/127.28.244.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.129.keytab.
I20260929 20:29:48.119238 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.129:0
--local_ip_for_outbound_sockets=127.28.244.129
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:35517
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.129.keytab
--principal=oryx/127.28.244.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:34121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:29:48.480422 4850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:48.480727 4850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:48.480846 4850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:48.489346 4850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:48.489478 4850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:48.489573 4850 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:29:48.489655 4850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.129
I20260929 20:29:48.500285 4850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:34121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.129.keytab
--principal=oryx/127.28.244.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.244.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.244.190:35517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.129
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:48.501648 4850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:48.503006 4850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:48.517773 4855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:48.517869 4856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:48.517838 4858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:48.518998 4850 server_base.cc:1034] running on GCE node
I20260929 20:29:48.519619 4850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:48.520704 4850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:48.521943 4850 hybrid_clock.cc:648] HybridClock initialized: now 1790713788521844 us; error 68 us; skew 500 ppm
I20260929 20:29:48.522221 4850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:48 dist-test-slave-p29l krb5kdc[4734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713788, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.244.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:48.525808 4850 init.cc:377] Logged in from keytab as oryx/127.28.244.129@KRBTEST.COM (short username oryx)
I20260929 20:29:48.527331 4850 webserver.cc:460] Webserver started at http://127.28.244.129:37969/ using document root <none> and password file <none>
I20260929 20:29:48.528115 4850 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:48.528206 4850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:48.528524 4850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:48.531459 4850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-0/data/instance:
uuid: "c03e7c30b2ef4546aed0e56cc6f3c685"
format_stamp: "Formatted at 2026-09-29 20:29:48 on dist-test-slave-p29l"
server_key: "2e8447188bd2128d9732af2764698caa"
server_key_iv: "7bd703f4c75410f193ccf895da9f56b5"
server_key_version: "encryptionkey@0"
I20260929 20:29:48.532337 4850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance:
uuid: "c03e7c30b2ef4546aed0e56cc6f3c685"
format_stamp: "Formatted at 2026-09-29 20:29:48 on dist-test-slave-p29l"
server_key: "2e8447188bd2128d9732af2764698caa"
server_key_iv: "7bd703f4c75410f193ccf895da9f56b5"
server_key_version: "encryptionkey@0"
I20260929 20:29:48.537609 4850 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:29:48.541388 4865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:48.543035 4850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:29:48.543191 4850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-0/wal
uuid: "c03e7c30b2ef4546aed0e56cc6f3c685"
format_stamp: "Formatted at 2026-09-29 20:29:48 on dist-test-slave-p29l"
server_key: "2e8447188bd2128d9732af2764698caa"
server_key_iv: "7bd703f4c75410f193ccf895da9f56b5"
server_key_version: "encryptionkey@0"
I20260929 20:29:48.543394 4850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:48.562288 4850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:48.567088 4850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:48.567355 4850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:48.570367 4850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:29:48.570461 4850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:48.570628 4850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:29:48.570732 4850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:29:48 dist-test-slave-p29l krb5kdc[4734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713788, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.244.129@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM
I20260929 20:29:48.599308 4874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:48.570950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:37745 (local address 127.28.244.190:35517)
0929 20:29:48.571738 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:48.571745 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:48.572942 (+ 1197us) server_negotiation.cc:407] Connection header received
0929 20:29:48.575508 (+ 2566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:48.575514 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:48.575622 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:48.575756 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:48.578902 (+ 3146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:48.580090 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:48.584479 (+ 4389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:48.585346 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:48.585609 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:48.590388 (+ 4779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:48.590399 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:48.590419 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:48.590484 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:48.595463 (+ 4979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:48.596761 (+ 1298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:48.596766 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:48.596768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:48.596902 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:48.597393 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:48.597404 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:48.597407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:48.597689 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:48.597856 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:48.598656 (+ 800us) server_negotiation.cc:299] Negotiation successful
0929 20:29:48.598919 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":520,"threads_started":1}
I20260929 20:29:48.600847 4875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:48.571718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35517 (local address 127.28.244.129:37745)
0929 20:29:48.572758 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:48.572805 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:48.575141 (+ 2336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:48.576143 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:48.576162 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:48.577376 (+ 1214us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:48.578641 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:48.578679 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:48.580311 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:48.580319 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:48.584060 (+ 3741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:48.584079 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:48.585687 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:48.585693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:48.585916 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:48.586786 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:48.586813 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:48.590012 (+ 3199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:48.595720 (+ 5708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:48.595729 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:48.595744 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:48.596555 (+ 811us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:48.597029 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:48.597035 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:48.597039 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:48.597259 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:48.597828 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:48.597836 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:48.598246 (+ 410us) client_negotiation.cc:769] Sending connection context
0929 20:29:48.598546 (+ 300us) client_negotiation.cc:240] Negotiation successful
0929 20:29:48.599115 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"thread_start_us":420,"threads_started":1}
I20260929 20:29:48.625550 4874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:48.607957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:33155 (local address 127.28.244.190:35517)
0929 20:29:48.608202 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:48.608212 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:48.609023 (+ 811us) server_negotiation.cc:407] Connection header received
0929 20:29:48.609201 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:48.609211 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:48.609328 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:48.609473 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:48.610519 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:48.612026 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:48.614296 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:48.615125 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:48.615317 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:48.618066 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:48.618076 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:48.618084 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:48.618139 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:48.622396 (+ 4257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:48.623259 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:48.623267 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:48.623270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:48.623408 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:48.623942 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:48.623948 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:48.623952 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:48.624300 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:48.624480 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:48.625027 (+ 547us) server_negotiation.cc:299] Negotiation successful
0929 20:29:48.625207 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 20:29:48.625948 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:48.608006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35517 (local address 127.28.244.129:33155)
0929 20:29:48.608881 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:48.608902 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:48.609044 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:48.609524 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:48.609528 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:48.610041 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:48.610357 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:48.610367 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:48.612286 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:48.612291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:48.614109 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:48.614131 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:48.615400 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:48.615414 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:48.615671 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:48.616544 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:48.616573 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:48.617831 (+ 1258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:48.622555 (+ 4724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:48.622563 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:48.622571 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:48.623117 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:48.623527 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:48.623534 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:48.623539 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:48.623762 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:48.624460 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:48.624469 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:48.624705 (+ 236us) client_negotiation.cc:769] Sending connection context
0929 20:29:48.624897 (+ 192us) client_negotiation.cc:240] Negotiation successful
0929 20:29:48.625335 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":432,"threads_started":1}
I20260929 20:29:48.638871 4850 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.129:45851
I20260929 20:29:48.638890 4985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.129:45851 every 8 connection(s)
I20260929 20:29:48.640349 4850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
I20260929 20:29:48.642701 4986 heartbeater.cc:347] Connected to a master server at 127.28.244.190:35517
I20260929 20:29:48.642937 4986 heartbeater.cc:464] Registering TS with master...
I20260929 20:29:48.643586 4986 heartbeater.cc:511] Master 127.28.244.190:35517 requested a full tablet report, sending...
I20260929 20:29:48.644443 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 4850
I20260929 20:29:48.644661 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance
I20260929 20:29:48.645303 29650 external_mini_cluster.cc:1428] Setting key 04ae6d32a1f838a7bd18850d4e43a680
I20260929 20:29:48.645906 4779 ts_manager.cc:194] Registered new tserver with Master: c03e7c30b2ef4546aed0e56cc6f3c685 (127.28.244.129:45851)
I20260929 20:29:48.647991 4779 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.244.129@KRBTEST.COM'} at 127.28.244.129:37745
WARNING: no policy specified for oryx/127.28.244.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.244.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.244.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.130.keytab.
Entry for principal oryx/127.28.244.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.130.keytab.
I20260929 20:29:48.695611 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.130:0
--local_ip_for_outbound_sockets=127.28.244.130
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:35517
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.130.keytab
--principal=oryx/127.28.244.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:34121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:29:49.030010 4990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:49.030288 4990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:49.030383 4990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:49.037655 4990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:49.037770 4990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:49.037859 4990 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:29:49.037941 4990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.130
I20260929 20:29:49.048811 4990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:34121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.130.keytab
--principal=oryx/127.28.244.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.244.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.244.190:35517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.130
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:49.050094 4990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:49.051405 4990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:49.065073 4996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:49.065095 4995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:49.065277 4998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:49.066078 4990 server_base.cc:1034] running on GCE node
I20260929 20:29:49.066776 4990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:49.067874 4990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:49.069159 4990 hybrid_clock.cc:648] HybridClock initialized: now 1790713789069071 us; error 72 us; skew 500 ppm
I20260929 20:29:49.069519 4990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:49 dist-test-slave-p29l krb5kdc[4734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713789, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.244.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:49.073093 4990 init.cc:377] Logged in from keytab as oryx/127.28.244.130@KRBTEST.COM (short username oryx)
I20260929 20:29:49.074715 4990 webserver.cc:460] Webserver started at http://127.28.244.130:46741/ using document root <none> and password file <none>
I20260929 20:29:49.075495 4990 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:49.075639 4990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:49.076040 4990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:49.079051 4990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-1/data/instance:
uuid: "d7103c3c79fe47f593e3c15745dd7ed7"
format_stamp: "Formatted at 2026-09-29 20:29:49 on dist-test-slave-p29l"
server_key: "89c7cbd1105331086595d86504801198"
server_key_iv: "42473718261c3d1dc4ecc0501da4d327"
server_key_version: "encryptionkey@0"
I20260929 20:29:49.079975 4990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance:
uuid: "d7103c3c79fe47f593e3c15745dd7ed7"
format_stamp: "Formatted at 2026-09-29 20:29:49 on dist-test-slave-p29l"
server_key: "89c7cbd1105331086595d86504801198"
server_key_iv: "42473718261c3d1dc4ecc0501da4d327"
server_key_version: "encryptionkey@0"
I20260929 20:29:49.085331 4990 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 20:29:49.089126 5005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:49.090540 4990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 20:29:49.090746 4990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-1/wal
uuid: "d7103c3c79fe47f593e3c15745dd7ed7"
format_stamp: "Formatted at 2026-09-29 20:29:49 on dist-test-slave-p29l"
server_key: "89c7cbd1105331086595d86504801198"
server_key_iv: "42473718261c3d1dc4ecc0501da4d327"
server_key_version: "encryptionkey@0"
I20260929 20:29:49.091013 4990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:49.106792 4990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:49.111331 4990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:49.111634 4990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:49.114529 4990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:29:49.114681 4990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:49.114835 4990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:29:49.114951 4990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:29:49 dist-test-slave-p29l krb5kdc[4734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713789, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.244.130@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM
I20260929 20:29:49.142208 4874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:49.115626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:34793 (local address 127.28.244.190:35517)
0929 20:29:49.115797 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:49.115854 (+ 57us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:49.117393 (+ 1539us) server_negotiation.cc:407] Connection header received
0929 20:29:49.120180 (+ 2787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:49.120185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:49.120264 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:49.120365 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:49.123268 (+ 2903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.124437 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.127191 (+ 2754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.127806 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.127984 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:49.134592 (+ 6608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:49.134602 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:49.134611 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:49.134666 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:49.138454 (+ 3788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:49.139366 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:49.139371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:49.139375 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:49.139494 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:49.140102 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:49.140108 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:49.140114 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:49.140436 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:49.140604 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:49.141591 (+ 987us) server_negotiation.cc:299] Negotiation successful
0929 20:29:49.141824 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 20:29:49.143447 5014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:49.116323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35517 (local address 127.28.244.130:34793)
0929 20:29:49.117213 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:49.117253 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:49.119723 (+ 2470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:49.121037 (+ 1314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:49.121054 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:49.121970 (+ 916us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:49.123027 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:49.123070 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.124704 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.124713 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:49.126912 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:49.126950 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.128151 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.128162 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:49.128599 (+ 437us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:49.129596 (+ 997us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:49.129648 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:49.134243 (+ 4595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:49.138628 (+ 4385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:49.138638 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:49.138659 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:49.139137 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:49.139624 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:49.139629 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:49.139633 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:49.139926 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:49.140582 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:49.140594 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:49.141133 (+ 539us) client_negotiation.cc:769] Sending connection context
0929 20:29:49.141485 (+ 352us) client_negotiation.cc:240] Negotiation successful
0929 20:29:49.142105 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":401,"threads_started":1}
I20260929 20:29:49.163726 4990 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.130:45127
I20260929 20:29:49.163758 5124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.130:45127 every 8 connection(s)
I20260929 20:29:49.165386 4990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
I20260929 20:29:49.168030 4874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:49.150399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:40935 (local address 127.28.244.190:35517)
0929 20:29:49.150605 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:49.150614 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:49.151263 (+ 649us) server_negotiation.cc:407] Connection header received
0929 20:29:49.151482 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:49.151485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:49.151548 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:49.151632 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:49.153409 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.154365 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.156922 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.157750 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.157894 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:49.161173 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:49.161181 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:49.161185 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:49.161229 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:49.164745 (+ 3516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:49.165498 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:49.165503 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:49.165507 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:49.165680 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:49.166145 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:49.166150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:49.166154 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:49.166678 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:49.166830 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:49.167444 (+ 614us) server_negotiation.cc:299] Negotiation successful
0929 20:29:49.167675 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:29:49.168074 5125 heartbeater.cc:347] Connected to a master server at 127.28.244.190:35517
I20260929 20:29:49.168134 5094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:49.150407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35517 (local address 127.28.244.130:40935)
0929 20:29:49.151132 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:49.151158 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:49.151342 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:49.151699 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:49.151707 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:49.152558 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:49.153210 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:49.153231 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.154529 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.154535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:49.156685 (+ 2150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:49.156708 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.158211 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.158221 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:49.158514 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:49.159476 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:49.159512 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:49.160924 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:49.164917 (+ 3993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:49.164922 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:49.164925 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:49.165336 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:49.165838 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:49.165842 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:49.165844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:49.165986 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:49.166910 (+ 924us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:49.166919 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:49.167134 (+ 215us) client_negotiation.cc:769] Sending connection context
0929 20:29:49.167299 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 20:29:49.167612 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":347,"threads_started":1}
I20260929 20:29:49.168325 5125 heartbeater.cc:464] Registering TS with master...
I20260929 20:29:49.169023 5125 heartbeater.cc:511] Master 127.28.244.190:35517 requested a full tablet report, sending...
I20260929 20:29:49.170163 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 4990
I20260929 20:29:49.170346 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance
I20260929 20:29:49.170601 4779 ts_manager.cc:194] Registered new tserver with Master: d7103c3c79fe47f593e3c15745dd7ed7 (127.28.244.130:45127)
I20260929 20:29:49.170939 29650 external_mini_cluster.cc:1428] Setting key a3ede1fb3a791b224fbff24f2eaa3bb2
I20260929 20:29:49.171751 4779 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.244.130@KRBTEST.COM'} at 127.28.244.130:34793
WARNING: no policy specified for oryx/127.28.244.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.244.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.244.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.131.keytab.
Entry for principal oryx/127.28.244.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.131.keytab.
I20260929 20:29:49.222940 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.131:0
--local_ip_for_outbound_sockets=127.28.244.131
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:35517
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.131.keytab
--principal=oryx/127.28.244.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:34121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:29:49.559747 5129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:49.560113 5129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:49.560276 5129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:49.567682 5129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:49.567821 5129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:49.567909 5129 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:29:49.567982 5129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.131
I20260929 20:29:49.578243 5129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:34121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.131.keytab
--principal=oryx/127.28.244.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.244.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.244.190:35517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.131
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:49.579607 5129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:49.581110 5129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:49.595418 5134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:49.595270 5135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:49.595443 5137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:49.596561 5129 server_base.cc:1034] running on GCE node
I20260929 20:29:49.597229 5129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:49.598260 5129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:49.599545 5129 hybrid_clock.cc:648] HybridClock initialized: now 1790713789599452 us; error 64 us; skew 500 ppm
I20260929 20:29:49.599928 5129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:49 dist-test-slave-p29l krb5kdc[4734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713789, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.244.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:49.603106 5129 init.cc:377] Logged in from keytab as oryx/127.28.244.131@KRBTEST.COM (short username oryx)
I20260929 20:29:49.604796 5129 webserver.cc:460] Webserver started at http://127.28.244.131:33417/ using document root <none> and password file <none>
I20260929 20:29:49.605523 5129 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:49.605613 5129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:49.605931 5129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:49.608968 5129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-2/data/instance:
uuid: "3c868738424b42cc824b67b80bd5b861"
format_stamp: "Formatted at 2026-09-29 20:29:49 on dist-test-slave-p29l"
server_key: "10113f8e185b775de8d3a4bcf1fbe3b4"
server_key_iv: "7f80735e25c20367574405c78923dd64"
server_key_version: "encryptionkey@0"
I20260929 20:29:49.609753 5129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance:
uuid: "3c868738424b42cc824b67b80bd5b861"
format_stamp: "Formatted at 2026-09-29 20:29:49 on dist-test-slave-p29l"
server_key: "10113f8e185b775de8d3a4bcf1fbe3b4"
server_key_iv: "7f80735e25c20367574405c78923dd64"
server_key_version: "encryptionkey@0"
I20260929 20:29:49.615283 5129 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:29:49.619098 5144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:49.620580 5129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:29:49.620761 5129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-2/wal
uuid: "3c868738424b42cc824b67b80bd5b861"
format_stamp: "Formatted at 2026-09-29 20:29:49 on dist-test-slave-p29l"
server_key: "10113f8e185b775de8d3a4bcf1fbe3b4"
server_key_iv: "7f80735e25c20367574405c78923dd64"
server_key_version: "encryptionkey@0"
I20260929 20:29:49.620986 5129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:49.637038 5129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:49.641692 5129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:49.641983 5129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:49.645277 5129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:29:49.645545 5129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:49.645730 5129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:29:49.645844 5129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260929 20:29:49.651028 4986 heartbeater.cc:503] Master 127.28.244.190:35517 was elected leader, sending a full tablet report...
Sep 29 20:29:49 dist-test-slave-p29l krb5kdc[4734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713789, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.244.131@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM
I20260929 20:29:49.670113 5153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:49.646540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35517 (local address 127.28.244.131:39053)
0929 20:29:49.647454 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:49.647499 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:49.649470 (+ 1971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:49.650264 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:49.650276 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:49.651180 (+ 904us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:49.652157 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:49.652185 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.653940 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.653948 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:49.655350 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:49.655363 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.656651 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.656661 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:49.656949 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:49.657909 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:49.657957 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:49.662087 (+ 4130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:49.665899 (+ 3812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:49.665916 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:49.665937 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:49.666397 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:49.666800 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:49.666805 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:49.666809 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:49.667028 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:49.667661 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:49.667669 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:49.668137 (+ 468us) client_negotiation.cc:769] Sending connection context
0929 20:29:49.668545 (+ 408us) client_negotiation.cc:240] Negotiation successful
0929 20:29:49.668982 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":377,"threads_started":1}
I20260929 20:29:49.671329 4874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:49.645844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:39053 (local address 127.28.244.190:35517)
0929 20:29:49.646070 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:49.646077 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:49.647692 (+ 1615us) server_negotiation.cc:407] Connection header received
0929 20:29:49.649788 (+ 2096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:49.649794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:49.649887 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:49.650006 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:49.652376 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.653750 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.655512 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.656354 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.656515 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:49.662391 (+ 5876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:49.662396 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:49.662400 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:49.662450 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:49.665740 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:49.666535 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:49.666540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:49.666544 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:49.666686 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:49.667149 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:49.667155 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:49.667158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:49.667528 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:49.667690 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:49.670915 (+ 3225us) server_negotiation.cc:299] Negotiation successful
0929 20:29:49.671094 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 20:29:49.693745 4874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:49.677832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:34763 (local address 127.28.244.190:35517)
0929 20:29:49.678060 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:49.678070 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:49.678728 (+ 658us) server_negotiation.cc:407] Connection header received
0929 20:29:49.678942 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:49.678945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:49.679008 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:49.679092 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:49.680716 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.681791 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.683491 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.684009 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.684151 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:49.686076 (+ 1925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:49.686081 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:49.686085 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:49.686117 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:49.690195 (+ 4078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:49.691309 (+ 1114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:49.691317 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:49.691322 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:49.691497 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:49.692053 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:49.692057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:49.692060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:49.692441 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:49.692620 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:49.693131 (+ 511us) server_negotiation.cc:299] Negotiation successful
0929 20:29:49.693330 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 20:29:49.694056 5212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:49.677811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35517 (local address 127.28.244.131:34763)
0929 20:29:49.678588 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:49.678621 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:49.678808 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:49.679164 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:49.679172 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:49.679952 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:49.680513 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:49.680539 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.681979 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.681983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:49.683273 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:49.683286 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.684232 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.684238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:49.684395 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:49.685041 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:49.685058 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:49.685880 (+ 822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:49.690466 (+ 4586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:49.690472 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:49.690477 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:49.691112 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:49.691625 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:49.691632 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:49.691637 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:49.691874 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:49.692587 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:49.692592 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:49.692820 (+ 228us) client_negotiation.cc:769] Sending connection context
0929 20:29:49.693014 (+ 194us) client_negotiation.cc:240] Negotiation successful
0929 20:29:49.693446 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":394,"threads_started":1}
I20260929 20:29:49.708820 5129 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.131:44831
I20260929 20:29:49.708854 5263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.131:44831 every 8 connection(s)
I20260929 20:29:49.710103 5129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
I20260929 20:29:49.712387 5264 heartbeater.cc:347] Connected to a master server at 127.28.244.190:35517
I20260929 20:29:49.712692 5264 heartbeater.cc:464] Registering TS with master...
I20260929 20:29:49.713413 5264 heartbeater.cc:511] Master 127.28.244.190:35517 requested a full tablet report, sending...
I20260929 20:29:49.715139 4779 ts_manager.cc:194] Registered new tserver with Master: 3c868738424b42cc824b67b80bd5b861 (127.28.244.131:44831)
I20260929 20:29:49.716150 4779 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.244.131@KRBTEST.COM'} at 127.28.244.131:39053
I20260929 20:29:49.716364 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 5129
I20260929 20:29:49.716559 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance
I20260929 20:29:49.717092 29650 external_mini_cluster.cc:1428] Setting key 3a3b15a432715d77c2f98e96dbd1c99e
I20260929 20:29:49.720153 29650 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 20:29:49 dist-test-slave-p29l krb5kdc[4734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713789, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 20:29:49 dist-test-slave-p29l krb5kdc[4734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:29:49 dist-test-slave-p29l krb5kdc[4734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 20:29:49.772938 4874 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:29:49.761866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47566 (local address 127.28.244.190:35517)
0929 20:29:49.762091 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:49.762099 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:49.762282 (+ 183us) server_negotiation.cc:407] Connection header received
0929 20:29:49.762478 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:49.762483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:49.762555 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:49.762653 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:49.764277 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.765085 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.766371 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.766841 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.766952 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:49.772707 (+ 5755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47566: BlockingRecv error: recv got EOF from 127.0.0.1:47566 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 29 20:29:49 dist-test-slave-p29l krb5kdc[4734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713789, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM
I20260929 20:29:49.790664 4874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:49.776231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47580 (local address 127.28.244.190:35517)
0929 20:29:49.776442 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:49.776448 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:49.776655 (+ 207us) server_negotiation.cc:407] Connection header received
0929 20:29:49.776874 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:49.776879 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:49.776975 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:49.777081 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:49.778229 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.779006 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.780326 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.780760 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.780915 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:49.784886 (+ 3971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:49.784898 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:49.784901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:49.784940 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:49.787728 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:49.788510 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:49.788518 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:49.788527 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:49.788652 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:49.789141 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:49.789146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:49.789148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:49.789504 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:49.789746 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:49.790087 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 20:29:49.790276 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:29:49.795210 4779 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:29:49.802500 4779 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:29:49.834412 5286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:49.818092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:45851 (local address 127.0.0.1:34000)
0929 20:29:49.824301 (+ 6209us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:49.824327 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:49.824690 (+ 363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:49.825962 (+ 1272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:49.825974 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:49.826016 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:29:49.826595 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:49.826612 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.829295 (+ 2683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.829299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:49.830782 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:49.830796 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.832855 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.832864 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:49.833212 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:49.833293 (+ 81us) client_negotiation.cc:769] Sending connection context
0929 20:29:49.833480 (+ 187us) client_negotiation.cc:240] Negotiation successful
0929 20:29:49.833804 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6063,"thread_start_us":10082,"threads_started":1}
I20260929 20:29:49.835166 5285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:49.818678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34000 (local address 127.28.244.129:45851)
0929 20:29:49.819261 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:49.819270 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:49.824532 (+ 5262us) server_negotiation.cc:407] Connection header received
0929 20:29:49.824966 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:49.824980 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:49.825338 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:49.825560 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:29:49.827957 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.829144 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.830949 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.832598 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.832772 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:49.832966 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:49.834696 (+ 1730us) server_negotiation.cc:299] Negotiation successful
0929 20:29:49.834891 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":541,"threads_started":1}
I20260929 20:29:49.837894 5282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:49.817748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:45127 (local address 127.0.0.1:59090)
0929 20:29:49.818483 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:49.818532 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:49.818704 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:49.831401 (+ 12697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:49.831409 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:49.831442 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:29:49.831874 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:49.831886 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.833502 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.833510 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:49.835103 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:49.835116 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.837108 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.837120 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:49.837293 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:49.837330 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 20:29:49.837421 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 20:29:49.837573 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":727,"threads_started":1}
I20260929 20:29:49.838682 5284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:49.818293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59090 (local address 127.28.244.130:45127)
0929 20:29:49.827294 (+ 9001us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:49.827306 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:49.827339 (+ 33us) server_negotiation.cc:407] Connection header received
0929 20:29:49.827454 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:49.827465 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:49.827758 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:49.831169 (+ 3411us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:29:49.832084 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.833367 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.835362 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.836559 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.836726 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:49.836961 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:49.838271 (+ 1310us) server_negotiation.cc:299] Negotiation successful
0929 20:29:49.838418 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8766,"thread_start_us":9420,"threads_started":1}
I20260929 20:29:49.840019 4916 tablet_service.cc:1467] Processing CreateTablet for tablet f1432219d41b4faa8c2c5b259f35759c (DEFAULT_TABLE table=test-table [id=9129edbfa8d24ba1840fff2cecd9a470]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:29:49.841655 4916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1432219d41b4faa8c2c5b259f35759c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:49.842662 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:49.819505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55622 (local address 127.28.244.131:44831)
0929 20:29:49.834853 (+ 15348us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:49.834862 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:49.834898 (+ 36us) server_negotiation.cc:407] Connection header received
0929 20:29:49.835004 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:49.835014 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:49.835323 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:49.836439 (+ 1116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:29:49.836858 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.838477 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.840594 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.841609 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.841824 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:49.842010 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:49.842203 (+ 193us) server_negotiation.cc:299] Negotiation successful
0929 20:29:49.842352 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15141,"thread_start_us":15971,"threads_started":1}
I20260929 20:29:49.842533 5055 tablet_service.cc:1467] Processing CreateTablet for tablet f1432219d41b4faa8c2c5b259f35759c (DEFAULT_TABLE table=test-table [id=9129edbfa8d24ba1840fff2cecd9a470]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:29:49.843180 5283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:49.819115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:44831 (local address 127.0.0.1:55622)
0929 20:29:49.828554 (+ 9439us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:49.828581 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:49.828726 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:49.835790 (+ 7064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:49.835797 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:49.835887 (+ 90us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:29:49.836206 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:49.836217 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.838633 (+ 2416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.838639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:49.840415 (+ 1776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:49.840429 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.841821 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.841827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:49.841988 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:49.842020 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 20:29:49.842139 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 20:29:49.842280 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9256,"thread_start_us":4913,"threads_started":1}
I20260929 20:29:49.844295 5055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1432219d41b4faa8c2c5b259f35759c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:49.847033 5194 tablet_service.cc:1467] Processing CreateTablet for tablet f1432219d41b4faa8c2c5b259f35759c (DEFAULT_TABLE table=test-table [id=9129edbfa8d24ba1840fff2cecd9a470]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:29:49.848434 5194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1432219d41b4faa8c2c5b259f35759c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:49.857533 5288 tablet_bootstrap.cc:492] T f1432219d41b4faa8c2c5b259f35759c P c03e7c30b2ef4546aed0e56cc6f3c685: Bootstrap starting.
I20260929 20:29:49.860548 5290 tablet_bootstrap.cc:492] T f1432219d41b4faa8c2c5b259f35759c P 3c868738424b42cc824b67b80bd5b861: Bootstrap starting.
I20260929 20:29:49.863117 5288 tablet_bootstrap.cc:654] T f1432219d41b4faa8c2c5b259f35759c P c03e7c30b2ef4546aed0e56cc6f3c685: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:49.864503 5288 log.cc:824] T f1432219d41b4faa8c2c5b259f35759c P c03e7c30b2ef4546aed0e56cc6f3c685: Log is configured to *not* fsync() on all Append() calls
I20260929 20:29:49.864583 5289 tablet_bootstrap.cc:492] T f1432219d41b4faa8c2c5b259f35759c P d7103c3c79fe47f593e3c15745dd7ed7: Bootstrap starting.
I20260929 20:29:49.865099 5290 tablet_bootstrap.cc:654] T f1432219d41b4faa8c2c5b259f35759c P 3c868738424b42cc824b67b80bd5b861: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:49.866202 5290 log.cc:824] T f1432219d41b4faa8c2c5b259f35759c P 3c868738424b42cc824b67b80bd5b861: Log is configured to *not* fsync() on all Append() calls
I20260929 20:29:49.868513 5289 tablet_bootstrap.cc:654] T f1432219d41b4faa8c2c5b259f35759c P d7103c3c79fe47f593e3c15745dd7ed7: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:49.869310 5288 tablet_bootstrap.cc:492] T f1432219d41b4faa8c2c5b259f35759c P c03e7c30b2ef4546aed0e56cc6f3c685: No bootstrap required, opened a new log
I20260929 20:29:49.869562 5290 tablet_bootstrap.cc:492] T f1432219d41b4faa8c2c5b259f35759c P 3c868738424b42cc824b67b80bd5b861: No bootstrap required, opened a new log
I20260929 20:29:49.869599 5288 ts_tablet_manager.cc:1397] T f1432219d41b4faa8c2c5b259f35759c P c03e7c30b2ef4546aed0e56cc6f3c685: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260929 20:29:49.869726 5289 log.cc:824] T f1432219d41b4faa8c2c5b259f35759c P d7103c3c79fe47f593e3c15745dd7ed7: Log is configured to *not* fsync() on all Append() calls
I20260929 20:29:49.869834 5290 ts_tablet_manager.cc:1397] T f1432219d41b4faa8c2c5b259f35759c P 3c868738424b42cc824b67b80bd5b861: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260929 20:29:49.872831 5289 tablet_bootstrap.cc:492] T f1432219d41b4faa8c2c5b259f35759c P d7103c3c79fe47f593e3c15745dd7ed7: No bootstrap required, opened a new log
I20260929 20:29:49.873185 5289 ts_tablet_manager.cc:1397] T f1432219d41b4faa8c2c5b259f35759c P d7103c3c79fe47f593e3c15745dd7ed7: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 20:29:49.878051 5288 raft_consensus.cc:348] T f1432219d41b4faa8c2c5b259f35759c P c03e7c30b2ef4546aed0e56cc6f3c685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:49.878471 5288 raft_consensus.cc:374] T f1432219d41b4faa8c2c5b259f35759c P c03e7c30b2ef4546aed0e56cc6f3c685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:49.878618 5288 raft_consensus.cc:729] T f1432219d41b4faa8c2c5b259f35759c P c03e7c30b2ef4546aed0e56cc6f3c685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c03e7c30b2ef4546aed0e56cc6f3c685, State: Initialized, Role: FOLLOWER
I20260929 20:29:49.879283 5288 consensus_queue.cc:260] T f1432219d41b4faa8c2c5b259f35759c P c03e7c30b2ef4546aed0e56cc6f3c685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:49.880975 5288 ts_tablet_manager.cc:1428] T f1432219d41b4faa8c2c5b259f35759c P c03e7c30b2ef4546aed0e56cc6f3c685: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20260929 20:29:49.882761 5290 raft_consensus.cc:348] T f1432219d41b4faa8c2c5b259f35759c P 3c868738424b42cc824b67b80bd5b861 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:49.883301 5290 raft_consensus.cc:374] T f1432219d41b4faa8c2c5b259f35759c P 3c868738424b42cc824b67b80bd5b861 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:49.883424 5290 raft_consensus.cc:729] T f1432219d41b4faa8c2c5b259f35759c P 3c868738424b42cc824b67b80bd5b861 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c868738424b42cc824b67b80bd5b861, State: Initialized, Role: FOLLOWER
I20260929 20:29:49.883651 5289 raft_consensus.cc:348] T f1432219d41b4faa8c2c5b259f35759c P d7103c3c79fe47f593e3c15745dd7ed7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:49.884097 5289 raft_consensus.cc:374] T f1432219d41b4faa8c2c5b259f35759c P d7103c3c79fe47f593e3c15745dd7ed7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:49.884200 5289 raft_consensus.cc:729] T f1432219d41b4faa8c2c5b259f35759c P d7103c3c79fe47f593e3c15745dd7ed7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7103c3c79fe47f593e3c15745dd7ed7, State: Initialized, Role: FOLLOWER
I20260929 20:29:49.884109 5290 consensus_queue.cc:260] T f1432219d41b4faa8c2c5b259f35759c P 3c868738424b42cc824b67b80bd5b861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:49.884801 5289 consensus_queue.cc:260] T f1432219d41b4faa8c2c5b259f35759c P d7103c3c79fe47f593e3c15745dd7ed7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:49.886049 5290 ts_tablet_manager.cc:1428] T f1432219d41b4faa8c2c5b259f35759c P 3c868738424b42cc824b67b80bd5b861: Time spent starting tablet: real 0.016s user 0.007s sys 0.006s
I20260929 20:29:49.886660 5264 heartbeater.cc:503] Master 127.28.244.190:35517 was elected leader, sending a full tablet report...
I20260929 20:29:49.887332 5125 heartbeater.cc:503] Master 127.28.244.190:35517 was elected leader, sending a full tablet report...
I20260929 20:29:49.888011 5289 ts_tablet_manager.cc:1428] T f1432219d41b4faa8c2c5b259f35759c P d7103c3c79fe47f593e3c15745dd7ed7: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260929 20:29:49.892719 4987 tablet.cc:2369] T f1432219d41b4faa8c2c5b259f35759c P c03e7c30b2ef4546aed0e56cc6f3c685: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:29:49.917763 5126 tablet.cc:2369] T f1432219d41b4faa8c2c5b259f35759c P d7103c3c79fe47f593e3c15745dd7ed7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:29:49.926432 5295 raft_consensus.cc:482] T f1432219d41b4faa8c2c5b259f35759c P 3c868738424b42cc824b67b80bd5b861 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:29:49.926785 5295 raft_consensus.cc:504] T f1432219d41b4faa8c2c5b259f35759c P 3c868738424b42cc824b67b80bd5b861 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:49.928823 5295 leader_election.cc:288] T f1432219d41b4faa8c2c5b259f35759c P 3c868738424b42cc824b67b80bd5b861 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7103c3c79fe47f593e3c15745dd7ed7 (127.28.244.130:45127), c03e7c30b2ef4546aed0e56cc6f3c685 (127.28.244.129:45851)
I20260929 20:29:49.936192 5284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:49.929130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:33205 (local address 127.28.244.130:45127)
0929 20:29:49.929315 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:49.929321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:49.929528 (+ 207us) server_negotiation.cc:407] Connection header received
0929 20:29:49.929641 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:49.929650 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:49.929748 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:49.929858 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:29:49.930328 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.931307 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.933915 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.935126 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.935296 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:49.935377 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:49.935732 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 20:29:49.935920 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:29:49.936201 5153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:49.928984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:45127 (local address 127.28.244.131:33205)
0929 20:29:49.929288 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:49.929310 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:49.929461 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:49.929910 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:49.929915 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:49.929954 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:29:49.930191 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:49.930201 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.931562 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.931566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:49.933742 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:49.933758 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.935285 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.935289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:49.935549 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:49.935596 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 20:29:49.935695 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 20:29:49.935867 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":26}
I20260929 20:29:49.936374 5285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:49.929404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:49849 (local address 127.28.244.129:45851)
0929 20:29:49.929636 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:49.929644 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:49.930153 (+ 509us) server_negotiation.cc:407] Connection header received
0929 20:29:49.930341 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:49.930346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:49.930427 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:49.930528 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:29:49.930997 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.931924 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.934184 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.935386 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.935550 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:49.935601 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:49.935976 (+ 375us) server_negotiation.cc:299] Negotiation successful
0929 20:29:49.936107 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 20:29:49.936471 5297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:49.929112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:45851 (local address 127.28.244.131:49849)
0929 20:29:49.930046 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:49.930063 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:49.930190 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:49.930546 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:49.930549 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:49.930571 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:29:49.930837 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:49.930845 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.932048 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.932054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:49.934032 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:49.934042 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.935595 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.935600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:49.935769 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:49.935798 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 20:29:49.935938 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 20:29:49.936063 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"mutex_wait_us":4,"thread_start_us":575,"threads_started":1}
I20260929 20:29:49.937455 4941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1432219d41b4faa8c2c5b259f35759c" candidate_uuid: "3c868738424b42cc824b67b80bd5b861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" is_pre_election: true
I20260929 20:29:49.937502 5075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1432219d41b4faa8c2c5b259f35759c" candidate_uuid: "3c868738424b42cc824b67b80bd5b861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" is_pre_election: true
I20260929 20:29:49.937989 4941 raft_consensus.cc:2454] T f1432219d41b4faa8c2c5b259f35759c P c03e7c30b2ef4546aed0e56cc6f3c685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c868738424b42cc824b67b80bd5b861 in term 0.
I20260929 20:29:49.938004 5075 raft_consensus.cc:2454] T f1432219d41b4faa8c2c5b259f35759c P d7103c3c79fe47f593e3c15745dd7ed7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c868738424b42cc824b67b80bd5b861 in term 0.
I20260929 20:29:49.938687 5146 leader_election.cc:302] T f1432219d41b4faa8c2c5b259f35759c P 3c868738424b42cc824b67b80bd5b861 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3c868738424b42cc824b67b80bd5b861, c03e7c30b2ef4546aed0e56cc6f3c685, d7103c3c79fe47f593e3c15745dd7ed7; no voters:
I20260929 20:29:49.939139 5295 raft_consensus.cc:2781] T f1432219d41b4faa8c2c5b259f35759c P 3c868738424b42cc824b67b80bd5b861 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:29:49.939303 5295 raft_consensus.cc:482] T f1432219d41b4faa8c2c5b259f35759c P 3c868738424b42cc824b67b80bd5b861 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:29:49.939426 5295 raft_consensus.cc:3037] T f1432219d41b4faa8c2c5b259f35759c P 3c868738424b42cc824b67b80bd5b861 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:49.941735 5295 raft_consensus.cc:504] T f1432219d41b4faa8c2c5b259f35759c P 3c868738424b42cc824b67b80bd5b861 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:49.942406 5295 leader_election.cc:288] T f1432219d41b4faa8c2c5b259f35759c P 3c868738424b42cc824b67b80bd5b861 [CANDIDATE]: Term 1 election: Requested vote from peers d7103c3c79fe47f593e3c15745dd7ed7 (127.28.244.130:45127), c03e7c30b2ef4546aed0e56cc6f3c685 (127.28.244.129:45851)
I20260929 20:29:49.942915 5075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1432219d41b4faa8c2c5b259f35759c" candidate_uuid: "3c868738424b42cc824b67b80bd5b861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7103c3c79fe47f593e3c15745dd7ed7"
I20260929 20:29:49.943017 4941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1432219d41b4faa8c2c5b259f35759c" candidate_uuid: "3c868738424b42cc824b67b80bd5b861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685"
I20260929 20:29:49.943138 5075 raft_consensus.cc:3037] T f1432219d41b4faa8c2c5b259f35759c P d7103c3c79fe47f593e3c15745dd7ed7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:49.943274 4941 raft_consensus.cc:3037] T f1432219d41b4faa8c2c5b259f35759c P c03e7c30b2ef4546aed0e56cc6f3c685 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:49.945997 5075 raft_consensus.cc:2454] T f1432219d41b4faa8c2c5b259f35759c P d7103c3c79fe47f593e3c15745dd7ed7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c868738424b42cc824b67b80bd5b861 in term 1.
I20260929 20:29:49.946209 4941 raft_consensus.cc:2454] T f1432219d41b4faa8c2c5b259f35759c P c03e7c30b2ef4546aed0e56cc6f3c685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c868738424b42cc824b67b80bd5b861 in term 1.
I20260929 20:29:49.946473 5147 leader_election.cc:302] T f1432219d41b4faa8c2c5b259f35759c P 3c868738424b42cc824b67b80bd5b861 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c868738424b42cc824b67b80bd5b861, d7103c3c79fe47f593e3c15745dd7ed7; no voters:
I20260929 20:29:49.946792 5295 raft_consensus.cc:2781] T f1432219d41b4faa8c2c5b259f35759c P 3c868738424b42cc824b67b80bd5b861 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:29:49.947256 5295 raft_consensus.cc:686] T f1432219d41b4faa8c2c5b259f35759c P 3c868738424b42cc824b67b80bd5b861 [term 1 LEADER]: Becoming Leader. State: Replica: 3c868738424b42cc824b67b80bd5b861, State: Running, Role: LEADER
I20260929 20:29:49.947921 5295 consensus_queue.cc:237] T f1432219d41b4faa8c2c5b259f35759c P 3c868738424b42cc824b67b80bd5b861 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:49.952510 4779 catalog_manager.cc:5495] T f1432219d41b4faa8c2c5b259f35759c P 3c868738424b42cc824b67b80bd5b861 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c868738424b42cc824b67b80bd5b861 (127.28.244.131). New cstate: current_term: 1 leader_uuid: "3c868738424b42cc824b67b80bd5b861" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } health_report { overall_health: HEALTHY } } }
W20260929 20:29:49.961899 5265 tablet.cc:2369] T f1432219d41b4faa8c2c5b259f35759c P 3c868738424b42cc824b67b80bd5b861: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:29:49.988075 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:49.982641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55628 (local address 127.28.244.131:44831)
0929 20:29:49.982892 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:49.982898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:49.982917 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:29:49.983148 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:49.983152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:49.983233 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:49.983425 (+ 192us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:29:49.983778 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.984531 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.985868 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:49.986269 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:49.986411 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:49.986897 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:29:49.987449 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:29:49.987609 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:49.987699 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 20:29:49.987781 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 20:29:49.996536 4941 raft_consensus.cc:1261] T f1432219d41b4faa8c2c5b259f35759c P c03e7c30b2ef4546aed0e56cc6f3c685 [term 1 FOLLOWER]: Refusing update from remote peer 3c868738424b42cc824b67b80bd5b861: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:29:49.996543 5075 raft_consensus.cc:1261] T f1432219d41b4faa8c2c5b259f35759c P d7103c3c79fe47f593e3c15745dd7ed7 [term 1 FOLLOWER]: Refusing update from remote peer 3c868738424b42cc824b67b80bd5b861: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:29:49.997495 5298 consensus_queue.cc:1035] T f1432219d41b4faa8c2c5b259f35759c P 3c868738424b42cc824b67b80bd5b861 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:29:49.997900 5295 consensus_queue.cc:1035] T f1432219d41b4faa8c2c5b259f35759c P 3c868738424b42cc824b67b80bd5b861 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:29:50.011127 5303 mvcc.cc:205] Tried to move back new op lower bound from 7334763683815084032 to 7334763683630632960. Current Snapshot: MvccSnapshot[applied={T|T < 7334763683815084032}]
I20260929 20:29:50.026351 4778 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:29:50.026672 4778 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:29:50.030613 4778 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 535a1a0ea12c439ab74e69d63b55b5ea: Sending DeleteTablet for 3 replicas of tablet f1432219d41b4faa8c2c5b259f35759c
I20260929 20:29:50.031908 5055 tablet_service.cc:1514] Processing DeleteTablet for tablet f1432219d41b4faa8c2c5b259f35759c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:29:50 UTC) from {username='oryx', principal='oryx/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:59090
I20260929 20:29:50.031929 4916 tablet_service.cc:1514] Processing DeleteTablet for tablet f1432219d41b4faa8c2c5b259f35759c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:29:50 UTC) from {username='oryx', principal='oryx/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:34000
I20260929 20:29:50.031934 5194 tablet_service.cc:1514] Processing DeleteTablet for tablet f1432219d41b4faa8c2c5b259f35759c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:29:50 UTC) from {username='oryx', principal='oryx/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:55622
I20260929 20:29:50.034053 5313 tablet_replica.cc:330] T f1432219d41b4faa8c2c5b259f35759c P d7103c3c79fe47f593e3c15745dd7ed7: stopping tablet replica
I20260929 20:29:50.033912 4779 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 20:29:50.034574 5313 raft_consensus.cc:2229] T f1432219d41b4faa8c2c5b259f35759c P d7103c3c79fe47f593e3c15745dd7ed7 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260929 20:29:50.034821 4779 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:29:50.035099 5313 raft_consensus.cc:2258] T f1432219d41b4faa8c2c5b259f35759c P d7103c3c79fe47f593e3c15745dd7ed7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:29:50.037310 5313 ts_tablet_manager.cc:1905] T f1432219d41b4faa8c2c5b259f35759c P d7103c3c79fe47f593e3c15745dd7ed7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:29:50.046262 5055 tablet_service.cc:1467] Processing CreateTablet for tablet 1e889ba71a3d405a9bf9695f3a767947 (DEFAULT_TABLE table=test-table [id=61a58d13fd424ce79c7ccbe778267b0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:29:50.046370 5193 tablet_service.cc:1467] Processing CreateTablet for tablet 1e889ba71a3d405a9bf9695f3a767947 (DEFAULT_TABLE table=test-table [id=61a58d13fd424ce79c7ccbe778267b0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:29:50.046890 5055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e889ba71a3d405a9bf9695f3a767947. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:50.046998 5193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e889ba71a3d405a9bf9695f3a767947. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:50.052917 4915 tablet_service.cc:1467] Processing CreateTablet for tablet 1e889ba71a3d405a9bf9695f3a767947 (DEFAULT_TABLE table=test-table [id=61a58d13fd424ce79c7ccbe778267b0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:29:50.053580 4915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e889ba71a3d405a9bf9695f3a767947. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:50.061704 5289 tablet_bootstrap.cc:492] T 1e889ba71a3d405a9bf9695f3a767947 P d7103c3c79fe47f593e3c15745dd7ed7: Bootstrap starting.
I20260929 20:29:50.064066 5289 tablet_bootstrap.cc:654] T 1e889ba71a3d405a9bf9695f3a767947 P d7103c3c79fe47f593e3c15745dd7ed7: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:50.066787 5289 tablet_bootstrap.cc:492] T 1e889ba71a3d405a9bf9695f3a767947 P d7103c3c79fe47f593e3c15745dd7ed7: No bootstrap required, opened a new log
I20260929 20:29:50.066977 5289 ts_tablet_manager.cc:1397] T 1e889ba71a3d405a9bf9695f3a767947 P d7103c3c79fe47f593e3c15745dd7ed7: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260929 20:29:50.067555 5288 tablet_bootstrap.cc:492] T 1e889ba71a3d405a9bf9695f3a767947 P c03e7c30b2ef4546aed0e56cc6f3c685: Bootstrap starting.
I20260929 20:29:50.068105 5289 raft_consensus.cc:348] T 1e889ba71a3d405a9bf9695f3a767947 P d7103c3c79fe47f593e3c15745dd7ed7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:50.068410 5289 raft_consensus.cc:374] T 1e889ba71a3d405a9bf9695f3a767947 P d7103c3c79fe47f593e3c15745dd7ed7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:50.068509 5289 raft_consensus.cc:729] T 1e889ba71a3d405a9bf9695f3a767947 P d7103c3c79fe47f593e3c15745dd7ed7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7103c3c79fe47f593e3c15745dd7ed7, State: Initialized, Role: FOLLOWER
I20260929 20:29:50.068619 5290 tablet_bootstrap.cc:492] T 1e889ba71a3d405a9bf9695f3a767947 P 3c868738424b42cc824b67b80bd5b861: Bootstrap starting.
I20260929 20:29:50.068869 5289 consensus_queue.cc:260] T 1e889ba71a3d405a9bf9695f3a767947 P d7103c3c79fe47f593e3c15745dd7ed7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:50.069303 5288 tablet_bootstrap.cc:654] T 1e889ba71a3d405a9bf9695f3a767947 P c03e7c30b2ef4546aed0e56cc6f3c685: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:50.070426 5290 tablet_bootstrap.cc:654] T 1e889ba71a3d405a9bf9695f3a767947 P 3c868738424b42cc824b67b80bd5b861: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:50.070431 5289 ts_tablet_manager.cc:1428] T 1e889ba71a3d405a9bf9695f3a767947 P d7103c3c79fe47f593e3c15745dd7ed7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 20:29:50.074929 5311 tablet_replica.cc:330] T f1432219d41b4faa8c2c5b259f35759c P c03e7c30b2ef4546aed0e56cc6f3c685: stopping tablet replica
I20260929 20:29:50.075567 5311 raft_consensus.cc:2229] T f1432219d41b4faa8c2c5b259f35759c P c03e7c30b2ef4546aed0e56cc6f3c685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:29:50.076249 5311 raft_consensus.cc:2258] T f1432219d41b4faa8c2c5b259f35759c P c03e7c30b2ef4546aed0e56cc6f3c685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:29:50.079162 5311 ts_tablet_manager.cc:1905] T f1432219d41b4faa8c2c5b259f35759c P c03e7c30b2ef4546aed0e56cc6f3c685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:29:50.079911 5312 tablet_replica.cc:330] T f1432219d41b4faa8c2c5b259f35759c P 3c868738424b42cc824b67b80bd5b861: stopping tablet replica
I20260929 20:29:50.080369 5312 raft_consensus.cc:2229] T f1432219d41b4faa8c2c5b259f35759c P 3c868738424b42cc824b67b80bd5b861 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:29:50.081071 5312 raft_consensus.cc:2258] T f1432219d41b4faa8c2c5b259f35759c P 3c868738424b42cc824b67b80bd5b861 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:29:50.082415 5313 ts_tablet_manager.cc:1918] T f1432219d41b4faa8c2c5b259f35759c P d7103c3c79fe47f593e3c15745dd7ed7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:29:50.082644 5313 log.cc:1196] T f1432219d41b4faa8c2c5b259f35759c P d7103c3c79fe47f593e3c15745dd7ed7: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-1/wal/wals/f1432219d41b4faa8c2c5b259f35759c
I20260929 20:29:50.083354 5313 ts_tablet_manager.cc:1939] T f1432219d41b4faa8c2c5b259f35759c P d7103c3c79fe47f593e3c15745dd7ed7: Deleting consensus metadata
I20260929 20:29:50.085456 4765 catalog_manager.cc:4841] TS d7103c3c79fe47f593e3c15745dd7ed7 (127.28.244.130:45127): tablet f1432219d41b4faa8c2c5b259f35759c (table test-table [id=9129edbfa8d24ba1840fff2cecd9a470]) successfully deleted
I20260929 20:29:50.085561 5312 ts_tablet_manager.cc:1905] T f1432219d41b4faa8c2c5b259f35759c P 3c868738424b42cc824b67b80bd5b861: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:29:50.089241 5311 ts_tablet_manager.cc:1918] T f1432219d41b4faa8c2c5b259f35759c P c03e7c30b2ef4546aed0e56cc6f3c685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:29:50.089485 5311 log.cc:1196] T f1432219d41b4faa8c2c5b259f35759c P c03e7c30b2ef4546aed0e56cc6f3c685: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-0/wal/wals/f1432219d41b4faa8c2c5b259f35759c
I20260929 20:29:50.089731 5290 tablet_bootstrap.cc:492] T 1e889ba71a3d405a9bf9695f3a767947 P 3c868738424b42cc824b67b80bd5b861: No bootstrap required, opened a new log
I20260929 20:29:50.089967 5290 ts_tablet_manager.cc:1397] T 1e889ba71a3d405a9bf9695f3a767947 P 3c868738424b42cc824b67b80bd5b861: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.002s
I20260929 20:29:50.090134 5311 ts_tablet_manager.cc:1939] T f1432219d41b4faa8c2c5b259f35759c P c03e7c30b2ef4546aed0e56cc6f3c685: Deleting consensus metadata
I20260929 20:29:50.090960 5290 raft_consensus.cc:348] T 1e889ba71a3d405a9bf9695f3a767947 P 3c868738424b42cc824b67b80bd5b861 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:50.091297 5290 raft_consensus.cc:374] T 1e889ba71a3d405a9bf9695f3a767947 P 3c868738424b42cc824b67b80bd5b861 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:50.091450 5290 raft_consensus.cc:729] T 1e889ba71a3d405a9bf9695f3a767947 P 3c868738424b42cc824b67b80bd5b861 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c868738424b42cc824b67b80bd5b861, State: Initialized, Role: FOLLOWER
I20260929 20:29:50.091895 5290 consensus_queue.cc:260] T 1e889ba71a3d405a9bf9695f3a767947 P 3c868738424b42cc824b67b80bd5b861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:50.092103 4764 catalog_manager.cc:4841] TS c03e7c30b2ef4546aed0e56cc6f3c685 (127.28.244.129:45851): tablet f1432219d41b4faa8c2c5b259f35759c (table test-table [id=9129edbfa8d24ba1840fff2cecd9a470]) successfully deleted
I20260929 20:29:50.092622 5290 ts_tablet_manager.cc:1428] T 1e889ba71a3d405a9bf9695f3a767947 P 3c868738424b42cc824b67b80bd5b861: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 20:29:50.093829 5288 tablet_bootstrap.cc:492] T 1e889ba71a3d405a9bf9695f3a767947 P c03e7c30b2ef4546aed0e56cc6f3c685: No bootstrap required, opened a new log
I20260929 20:29:50.094007 5288 ts_tablet_manager.cc:1397] T 1e889ba71a3d405a9bf9695f3a767947 P c03e7c30b2ef4546aed0e56cc6f3c685: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.005s
I20260929 20:29:50.094026 5312 ts_tablet_manager.cc:1918] T f1432219d41b4faa8c2c5b259f35759c P 3c868738424b42cc824b67b80bd5b861: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:29:50.094190 5312 log.cc:1196] T f1432219d41b4faa8c2c5b259f35759c P 3c868738424b42cc824b67b80bd5b861: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-2/wal/wals/f1432219d41b4faa8c2c5b259f35759c
I20260929 20:29:50.094776 5312 ts_tablet_manager.cc:1939] T f1432219d41b4faa8c2c5b259f35759c P 3c868738424b42cc824b67b80bd5b861: Deleting consensus metadata
I20260929 20:29:50.094834 5288 raft_consensus.cc:348] T 1e889ba71a3d405a9bf9695f3a767947 P c03e7c30b2ef4546aed0e56cc6f3c685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:50.095063 5288 raft_consensus.cc:374] T 1e889ba71a3d405a9bf9695f3a767947 P c03e7c30b2ef4546aed0e56cc6f3c685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:50.095152 5288 raft_consensus.cc:729] T 1e889ba71a3d405a9bf9695f3a767947 P c03e7c30b2ef4546aed0e56cc6f3c685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c03e7c30b2ef4546aed0e56cc6f3c685, State: Initialized, Role: FOLLOWER
I20260929 20:29:50.095412 5288 consensus_queue.cc:260] T 1e889ba71a3d405a9bf9695f3a767947 P c03e7c30b2ef4546aed0e56cc6f3c685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:50.096621 4766 catalog_manager.cc:4841] TS 3c868738424b42cc824b67b80bd5b861 (127.28.244.131:44831): tablet f1432219d41b4faa8c2c5b259f35759c (table test-table [id=9129edbfa8d24ba1840fff2cecd9a470]) successfully deleted
I20260929 20:29:50.097330 5288 ts_tablet_manager.cc:1428] T 1e889ba71a3d405a9bf9695f3a767947 P c03e7c30b2ef4546aed0e56cc6f3c685: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260929 20:29:50.289496 5296 raft_consensus.cc:482] T 1e889ba71a3d405a9bf9695f3a767947 P d7103c3c79fe47f593e3c15745dd7ed7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:29:50.289748 5296 raft_consensus.cc:504] T 1e889ba71a3d405a9bf9695f3a767947 P d7103c3c79fe47f593e3c15745dd7ed7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:50.291589 5296 leader_election.cc:288] T 1e889ba71a3d405a9bf9695f3a767947 P d7103c3c79fe47f593e3c15745dd7ed7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c03e7c30b2ef4546aed0e56cc6f3c685 (127.28.244.129:45851), 3c868738424b42cc824b67b80bd5b861 (127.28.244.131:44831)
I20260929 20:29:50.299710 5285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:50.292083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:58657 (local address 127.28.244.129:45851)
0929 20:29:50.292382 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:50.292390 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:50.292946 (+ 556us) server_negotiation.cc:407] Connection header received
0929 20:29:50.293250 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:50.293259 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:50.293348 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:50.293449 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:29:50.294028 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:50.295171 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:50.297902 (+ 2731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:50.298732 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:50.298936 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:50.299018 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:50.299294 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 20:29:50.299435 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260929 20:29:50.299823 5317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:50.292036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:45851 (local address 127.28.244.130:58657)
0929 20:29:50.292826 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:50.292846 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:50.293072 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:50.293474 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:50.293479 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:50.293506 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:29:50.293831 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:50.293843 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:50.295300 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:50.295305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:50.297721 (+ 2416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:50.297737 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:50.298970 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:50.298976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:50.299134 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:50.299158 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 20:29:50.299269 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 20:29:50.299407 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":458,"threads_started":1}
I20260929 20:29:50.300617 4941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e889ba71a3d405a9bf9695f3a767947" candidate_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" is_pre_election: true
I20260929 20:29:50.300830 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:50.292026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:54087 (local address 127.28.244.131:44831)
0929 20:29:50.292230 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:50.292238 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:50.293337 (+ 1099us) server_negotiation.cc:407] Connection header received
0929 20:29:50.293563 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:50.293571 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:50.293687 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:50.293741 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:29:50.294902 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:50.296048 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:50.297970 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:50.299199 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:50.299358 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:50.299437 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:50.300392 (+ 955us) server_negotiation.cc:299] Negotiation successful
0929 20:29:50.300535 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 20:29:50.300897 4941 raft_consensus.cc:2454] T 1e889ba71a3d405a9bf9695f3a767947 P c03e7c30b2ef4546aed0e56cc6f3c685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7103c3c79fe47f593e3c15745dd7ed7 in term 0.
I20260929 20:29:50.301294 5318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:50.292278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:44831 (local address 127.28.244.130:54087)
0929 20:29:50.293223 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:50.293239 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:50.293367 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:50.293904 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:50.293907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:50.293927 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:29:50.294693 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:50.294706 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:50.296283 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:50.296288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:50.297799 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:50.297831 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:50.299617 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:50.299626 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:50.300208 (+ 582us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:50.300243 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 20:29:50.300351 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 20:29:50.300498 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"thread_start_us":616,"threads_started":1}
I20260929 20:29:50.301405 5007 leader_election.cc:302] T 1e889ba71a3d405a9bf9695f3a767947 P d7103c3c79fe47f593e3c15745dd7ed7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c03e7c30b2ef4546aed0e56cc6f3c685, d7103c3c79fe47f593e3c15745dd7ed7; no voters:
I20260929 20:29:50.301848 5296 raft_consensus.cc:2781] T 1e889ba71a3d405a9bf9695f3a767947 P d7103c3c79fe47f593e3c15745dd7ed7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:29:50.301988 5296 raft_consensus.cc:482] T 1e889ba71a3d405a9bf9695f3a767947 P d7103c3c79fe47f593e3c15745dd7ed7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:29:50.302080 5296 raft_consensus.cc:3037] T 1e889ba71a3d405a9bf9695f3a767947 P d7103c3c79fe47f593e3c15745dd7ed7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:50.302155 5219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e889ba71a3d405a9bf9695f3a767947" candidate_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c868738424b42cc824b67b80bd5b861" is_pre_election: true
I20260929 20:29:50.302582 5219 raft_consensus.cc:2454] T 1e889ba71a3d405a9bf9695f3a767947 P 3c868738424b42cc824b67b80bd5b861 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7103c3c79fe47f593e3c15745dd7ed7 in term 0.
I20260929 20:29:50.304657 5296 raft_consensus.cc:504] T 1e889ba71a3d405a9bf9695f3a767947 P d7103c3c79fe47f593e3c15745dd7ed7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:50.305348 5296 leader_election.cc:288] T 1e889ba71a3d405a9bf9695f3a767947 P d7103c3c79fe47f593e3c15745dd7ed7 [CANDIDATE]: Term 1 election: Requested vote from peers c03e7c30b2ef4546aed0e56cc6f3c685 (127.28.244.129:45851), 3c868738424b42cc824b67b80bd5b861 (127.28.244.131:44831)
I20260929 20:29:50.305788 4941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e889ba71a3d405a9bf9695f3a767947" candidate_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685"
I20260929 20:29:50.305892 5219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e889ba71a3d405a9bf9695f3a767947" candidate_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c868738424b42cc824b67b80bd5b861"
I20260929 20:29:50.306022 4941 raft_consensus.cc:3037] T 1e889ba71a3d405a9bf9695f3a767947 P c03e7c30b2ef4546aed0e56cc6f3c685 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:50.306107 5219 raft_consensus.cc:3037] T 1e889ba71a3d405a9bf9695f3a767947 P 3c868738424b42cc824b67b80bd5b861 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:50.309146 4941 raft_consensus.cc:2454] T 1e889ba71a3d405a9bf9695f3a767947 P c03e7c30b2ef4546aed0e56cc6f3c685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7103c3c79fe47f593e3c15745dd7ed7 in term 1.
I20260929 20:29:50.309412 5219 raft_consensus.cc:2454] T 1e889ba71a3d405a9bf9695f3a767947 P 3c868738424b42cc824b67b80bd5b861 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7103c3c79fe47f593e3c15745dd7ed7 in term 1.
I20260929 20:29:50.309660 5007 leader_election.cc:302] T 1e889ba71a3d405a9bf9695f3a767947 P d7103c3c79fe47f593e3c15745dd7ed7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c03e7c30b2ef4546aed0e56cc6f3c685, d7103c3c79fe47f593e3c15745dd7ed7; no voters:
I20260929 20:29:50.309983 5296 raft_consensus.cc:2781] T 1e889ba71a3d405a9bf9695f3a767947 P d7103c3c79fe47f593e3c15745dd7ed7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:29:50.310509 5296 raft_consensus.cc:686] T 1e889ba71a3d405a9bf9695f3a767947 P d7103c3c79fe47f593e3c15745dd7ed7 [term 1 LEADER]: Becoming Leader. State: Replica: d7103c3c79fe47f593e3c15745dd7ed7, State: Running, Role: LEADER
I20260929 20:29:50.310917 5296 consensus_queue.cc:237] T 1e889ba71a3d405a9bf9695f3a767947 P d7103c3c79fe47f593e3c15745dd7ed7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:50.315071 4778 catalog_manager.cc:5495] T 1e889ba71a3d405a9bf9695f3a767947 P d7103c3c79fe47f593e3c15745dd7ed7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7103c3c79fe47f593e3c15745dd7ed7 (127.28.244.130). New cstate: current_term: 1 leader_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } health_report { overall_health: UNKNOWN } } }
Sep 29 20:29:50 dist-test-slave-p29l krb5kdc[4734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713787, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.244.190@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM
I20260929 20:29:50.406977 5327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:50.390476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47582 (local address 127.28.244.190:35517)
0929 20:29:50.391100 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:50.391106 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:50.391685 (+ 579us) server_negotiation.cc:407] Connection header received
0929 20:29:50.391926 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:50.391930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:50.391988 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:50.392082 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:50.393533 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:50.394344 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:50.396263 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:50.396783 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:50.396881 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:50.401015 (+ 4134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:50.401021 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:50.401024 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:50.401056 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:50.404122 (+ 3066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:50.404798 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:50.404803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:50.404807 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:50.404977 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:50.405328 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:50.405333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:50.405337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:50.405612 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:50.405787 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:50.406335 (+ 548us) server_negotiation.cc:299] Negotiation successful
0929 20:29:50.406696 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":102,"server-negotiator.queue_time_us":458,"thread_start_us":685,"threads_started":1}
I20260929 20:29:50.407003 5326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:50.390395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35517 (local address 127.0.0.1:47582)
0929 20:29:50.391559 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:50.391585 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:50.391749 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:50.392156 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:50.392162 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:50.393092 (+ 930us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:50.393366 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:50.393379 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:50.394511 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:50.394517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:50.396114 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:50.396129 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:50.396960 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:50.396968 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:50.397149 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:50.397866 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:50.397890 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:50.400781 (+ 2891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:50.404246 (+ 3465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:50.404262 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:50.404271 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:50.404674 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:50.405079 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:50.405085 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:50.405089 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:50.405217 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:50.405783 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:50.405793 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:50.406021 (+ 228us) client_negotiation.cc:769] Sending connection context
0929 20:29:50.406231 (+ 210us) client_negotiation.cc:240] Negotiation successful
0929 20:29:50.406561 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1018,"thread_start_us":860,"threads_started":1}
I20260929 20:29:50.412878 4778 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:47582:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 20:29:50.414208 4778 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:29:50.421924 5055 tablet_service.cc:1467] Processing CreateTablet for tablet 9ae589f6e03f4d5abf8db0d7c039927a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7660a1cf8cd9477086d6265ebe3ff261]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:29:50.421914 4916 tablet_service.cc:1467] Processing CreateTablet for tablet 9ae589f6e03f4d5abf8db0d7c039927a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7660a1cf8cd9477086d6265ebe3ff261]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:29:50.422546 4916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ae589f6e03f4d5abf8db0d7c039927a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:50.422545 5055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ae589f6e03f4d5abf8db0d7c039927a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:50.425494 5194 tablet_service.cc:1467] Processing CreateTablet for tablet 9ae589f6e03f4d5abf8db0d7c039927a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7660a1cf8cd9477086d6265ebe3ff261]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:29:50.426123 5194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ae589f6e03f4d5abf8db0d7c039927a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:50.429917 5288 tablet_bootstrap.cc:492] T 9ae589f6e03f4d5abf8db0d7c039927a P c03e7c30b2ef4546aed0e56cc6f3c685: Bootstrap starting.
I20260929 20:29:50.431041 5289 tablet_bootstrap.cc:492] T 9ae589f6e03f4d5abf8db0d7c039927a P d7103c3c79fe47f593e3c15745dd7ed7: Bootstrap starting.
I20260929 20:29:50.431675 5288 tablet_bootstrap.cc:654] T 9ae589f6e03f4d5abf8db0d7c039927a P c03e7c30b2ef4546aed0e56cc6f3c685: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:50.432905 5289 tablet_bootstrap.cc:654] T 9ae589f6e03f4d5abf8db0d7c039927a P d7103c3c79fe47f593e3c15745dd7ed7: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:50.435067 5290 tablet_bootstrap.cc:492] T 9ae589f6e03f4d5abf8db0d7c039927a P 3c868738424b42cc824b67b80bd5b861: Bootstrap starting.
I20260929 20:29:50.436873 5290 tablet_bootstrap.cc:654] T 9ae589f6e03f4d5abf8db0d7c039927a P 3c868738424b42cc824b67b80bd5b861: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:50.440474 5290 tablet_bootstrap.cc:492] T 9ae589f6e03f4d5abf8db0d7c039927a P 3c868738424b42cc824b67b80bd5b861: No bootstrap required, opened a new log
I20260929 20:29:50.440697 5290 ts_tablet_manager.cc:1397] T 9ae589f6e03f4d5abf8db0d7c039927a P 3c868738424b42cc824b67b80bd5b861: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260929 20:29:50.442507 5290 raft_consensus.cc:348] T 9ae589f6e03f4d5abf8db0d7c039927a P 3c868738424b42cc824b67b80bd5b861 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:50.442854 5290 raft_consensus.cc:374] T 9ae589f6e03f4d5abf8db0d7c039927a P 3c868738424b42cc824b67b80bd5b861 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:50.442988 5290 raft_consensus.cc:729] T 9ae589f6e03f4d5abf8db0d7c039927a P 3c868738424b42cc824b67b80bd5b861 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c868738424b42cc824b67b80bd5b861, State: Initialized, Role: FOLLOWER
I20260929 20:29:50.443264 5289 tablet_bootstrap.cc:492] T 9ae589f6e03f4d5abf8db0d7c039927a P d7103c3c79fe47f593e3c15745dd7ed7: No bootstrap required, opened a new log
I20260929 20:29:50.443522 5289 ts_tablet_manager.cc:1397] T 9ae589f6e03f4d5abf8db0d7c039927a P d7103c3c79fe47f593e3c15745dd7ed7: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260929 20:29:50.443357 5290 consensus_queue.cc:260] T 9ae589f6e03f4d5abf8db0d7c039927a P 3c868738424b42cc824b67b80bd5b861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:50.444000 5298 tablet_replica.cc:439] T 9ae589f6e03f4d5abf8db0d7c039927a P 3c868738424b42cc824b67b80bd5b861: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } } }
I20260929 20:29:50.444314 5298 tablet_replica.cc:442] T 9ae589f6e03f4d5abf8db0d7c039927a P 3c868738424b42cc824b67b80bd5b861: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:29:50.444563 5289 raft_consensus.cc:348] T 9ae589f6e03f4d5abf8db0d7c039927a P d7103c3c79fe47f593e3c15745dd7ed7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:50.445416 5289 raft_consensus.cc:374] T 9ae589f6e03f4d5abf8db0d7c039927a P d7103c3c79fe47f593e3c15745dd7ed7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:50.445549 5289 raft_consensus.cc:729] T 9ae589f6e03f4d5abf8db0d7c039927a P d7103c3c79fe47f593e3c15745dd7ed7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7103c3c79fe47f593e3c15745dd7ed7, State: Initialized, Role: FOLLOWER
I20260929 20:29:50.445981 5289 consensus_queue.cc:260] T 9ae589f6e03f4d5abf8db0d7c039927a P d7103c3c79fe47f593e3c15745dd7ed7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:50.446552 5296 tablet_replica.cc:439] T 9ae589f6e03f4d5abf8db0d7c039927a P d7103c3c79fe47f593e3c15745dd7ed7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } } }
I20260929 20:29:50.446676 5289 ts_tablet_manager.cc:1428] T 9ae589f6e03f4d5abf8db0d7c039927a P d7103c3c79fe47f593e3c15745dd7ed7: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 20:29:50.446842 5296 tablet_replica.cc:442] T 9ae589f6e03f4d5abf8db0d7c039927a P d7103c3c79fe47f593e3c15745dd7ed7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:29:50.447708 5298 raft_consensus.cc:482] T 9ae589f6e03f4d5abf8db0d7c039927a P 3c868738424b42cc824b67b80bd5b861 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:29:50.447925 5298 raft_consensus.cc:504] T 9ae589f6e03f4d5abf8db0d7c039927a P 3c868738424b42cc824b67b80bd5b861 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:50.448596 5298 leader_election.cc:288] T 9ae589f6e03f4d5abf8db0d7c039927a P 3c868738424b42cc824b67b80bd5b861 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c03e7c30b2ef4546aed0e56cc6f3c685 (127.28.244.129:45851), d7103c3c79fe47f593e3c15745dd7ed7 (127.28.244.130:45127)
I20260929 20:29:50.449223 5075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ae589f6e03f4d5abf8db0d7c039927a" candidate_uuid: "3c868738424b42cc824b67b80bd5b861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" is_pre_election: true
I20260929 20:29:50.449215 4941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ae589f6e03f4d5abf8db0d7c039927a" candidate_uuid: "3c868738424b42cc824b67b80bd5b861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" is_pre_election: true
I20260929 20:29:50.449465 5075 raft_consensus.cc:2454] T 9ae589f6e03f4d5abf8db0d7c039927a P d7103c3c79fe47f593e3c15745dd7ed7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c868738424b42cc824b67b80bd5b861 in term 0.
I20260929 20:29:50.449918 5147 leader_election.cc:302] T 9ae589f6e03f4d5abf8db0d7c039927a P 3c868738424b42cc824b67b80bd5b861 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c868738424b42cc824b67b80bd5b861, d7103c3c79fe47f593e3c15745dd7ed7; no voters:
W20260929 20:29:50.450321 5146 leader_election.cc:341] T 9ae589f6e03f4d5abf8db0d7c039927a P 3c868738424b42cc824b67b80bd5b861 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c03e7c30b2ef4546aed0e56cc6f3c685 (127.28.244.129:45851): Illegal state: must be running to vote when last-logged opid is not known
I20260929 20:29:50.451516 5298 raft_consensus.cc:2781] T 9ae589f6e03f4d5abf8db0d7c039927a P 3c868738424b42cc824b67b80bd5b861 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:29:50.451614 5298 raft_consensus.cc:482] T 9ae589f6e03f4d5abf8db0d7c039927a P 3c868738424b42cc824b67b80bd5b861 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:29:50.451707 5298 raft_consensus.cc:3037] T 9ae589f6e03f4d5abf8db0d7c039927a P 3c868738424b42cc824b67b80bd5b861 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:50.454700 5298 raft_consensus.cc:504] T 9ae589f6e03f4d5abf8db0d7c039927a P 3c868738424b42cc824b67b80bd5b861 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:50.455869 5075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ae589f6e03f4d5abf8db0d7c039927a" candidate_uuid: "3c868738424b42cc824b67b80bd5b861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7103c3c79fe47f593e3c15745dd7ed7"
I20260929 20:29:50.456087 5075 raft_consensus.cc:3037] T 9ae589f6e03f4d5abf8db0d7c039927a P d7103c3c79fe47f593e3c15745dd7ed7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:50.455982 4941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ae589f6e03f4d5abf8db0d7c039927a" candidate_uuid: "3c868738424b42cc824b67b80bd5b861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685"
W20260929 20:29:50.457412 5146 leader_election.cc:341] T 9ae589f6e03f4d5abf8db0d7c039927a P 3c868738424b42cc824b67b80bd5b861 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c03e7c30b2ef4546aed0e56cc6f3c685 (127.28.244.129:45851): Illegal state: must be running to vote when last-logged opid is not known
I20260929 20:29:50.459077 5075 raft_consensus.cc:2454] T 9ae589f6e03f4d5abf8db0d7c039927a P d7103c3c79fe47f593e3c15745dd7ed7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c868738424b42cc824b67b80bd5b861 in term 1.
I20260929 20:29:50.459641 5147 leader_election.cc:302] T 9ae589f6e03f4d5abf8db0d7c039927a P 3c868738424b42cc824b67b80bd5b861 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c868738424b42cc824b67b80bd5b861, d7103c3c79fe47f593e3c15745dd7ed7; no voters: c03e7c30b2ef4546aed0e56cc6f3c685
I20260929 20:29:50.459877 5298 leader_election.cc:288] T 9ae589f6e03f4d5abf8db0d7c039927a P 3c868738424b42cc824b67b80bd5b861 [CANDIDATE]: Term 1 election: Requested vote from peers c03e7c30b2ef4546aed0e56cc6f3c685 (127.28.244.129:45851), d7103c3c79fe47f593e3c15745dd7ed7 (127.28.244.130:45127)
I20260929 20:29:50.460116 5306 raft_consensus.cc:2781] T 9ae589f6e03f4d5abf8db0d7c039927a P 3c868738424b42cc824b67b80bd5b861 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:29:50.460343 5306 raft_consensus.cc:686] T 9ae589f6e03f4d5abf8db0d7c039927a P 3c868738424b42cc824b67b80bd5b861 [term 1 LEADER]: Becoming Leader. State: Replica: 3c868738424b42cc824b67b80bd5b861, State: Running, Role: LEADER
I20260929 20:29:50.460807 5306 consensus_queue.cc:237] T 9ae589f6e03f4d5abf8db0d7c039927a P 3c868738424b42cc824b67b80bd5b861 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:50.461128 5290 ts_tablet_manager.cc:1428] T 9ae589f6e03f4d5abf8db0d7c039927a P 3c868738424b42cc824b67b80bd5b861: Time spent starting tablet: real 0.020s user 0.000s sys 0.002s
I20260929 20:29:50.462821 5288 tablet_bootstrap.cc:492] T 9ae589f6e03f4d5abf8db0d7c039927a P c03e7c30b2ef4546aed0e56cc6f3c685: No bootstrap required, opened a new log
I20260929 20:29:50.463091 5288 ts_tablet_manager.cc:1397] T 9ae589f6e03f4d5abf8db0d7c039927a P c03e7c30b2ef4546aed0e56cc6f3c685: Time spent bootstrapping tablet: real 0.033s user 0.011s sys 0.019s
I20260929 20:29:50.463915 5288 raft_consensus.cc:348] T 9ae589f6e03f4d5abf8db0d7c039927a P c03e7c30b2ef4546aed0e56cc6f3c685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:50.464224 5288 raft_consensus.cc:374] T 9ae589f6e03f4d5abf8db0d7c039927a P c03e7c30b2ef4546aed0e56cc6f3c685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:50.464330 5288 raft_consensus.cc:729] T 9ae589f6e03f4d5abf8db0d7c039927a P c03e7c30b2ef4546aed0e56cc6f3c685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c03e7c30b2ef4546aed0e56cc6f3c685, State: Initialized, Role: FOLLOWER
I20260929 20:29:50.464485 5298 tablet_replica.cc:439] T 9ae589f6e03f4d5abf8db0d7c039927a P 3c868738424b42cc824b67b80bd5b861: TxnStatusTablet state changed. Reason: New leader 3c868738424b42cc824b67b80bd5b861. Latest consensus state: current_term: 1 leader_uuid: "3c868738424b42cc824b67b80bd5b861" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } } }
I20260929 20:29:50.464336 4778 catalog_manager.cc:5495] T 9ae589f6e03f4d5abf8db0d7c039927a P 3c868738424b42cc824b67b80bd5b861 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c868738424b42cc824b67b80bd5b861 (127.28.244.131). New cstate: current_term: 1 leader_uuid: "3c868738424b42cc824b67b80bd5b861" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } health_report { overall_health: HEALTHY } } }
I20260929 20:29:50.464742 5298 tablet_replica.cc:442] T 9ae589f6e03f4d5abf8db0d7c039927a P 3c868738424b42cc824b67b80bd5b861: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:29:50.464653 5288 consensus_queue.cc:260] T 9ae589f6e03f4d5abf8db0d7c039927a P c03e7c30b2ef4546aed0e56cc6f3c685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:50.465250 5294 tablet_replica.cc:439] T 9ae589f6e03f4d5abf8db0d7c039927a P c03e7c30b2ef4546aed0e56cc6f3c685: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } } }
I20260929 20:29:50.465737 5294 tablet_replica.cc:442] T 9ae589f6e03f4d5abf8db0d7c039927a P c03e7c30b2ef4546aed0e56cc6f3c685: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:29:50.466817 5288 ts_tablet_manager.cc:1428] T 9ae589f6e03f4d5abf8db0d7c039927a P c03e7c30b2ef4546aed0e56cc6f3c685: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260929 20:29:50.467739 5332 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 20:29:50.468047 5332 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 20:29:50.486338 5326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:50.480731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:44831 (local address 127.0.0.1:55640)
0929 20:29:50.480972 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:50.480999 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:50.481116 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:50.481478 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:50.481482 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:50.481491 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:29:50.481638 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:50.481645 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:50.483045 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:50.483049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:50.484181 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:50.484198 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:50.485184 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:50.485188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:50.485346 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:50.485409 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:29:50.485913 (+ 504us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:29:50.485931 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 20:29:50.486001 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 20:29:50.486121 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 20:29:50.486457 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:50.480732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55640 (local address 127.28.244.131:44831)
0929 20:29:50.480949 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:50.480955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:50.481072 (+ 117us) server_negotiation.cc:407] Connection header received
0929 20:29:50.481268 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:50.481273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:50.481353 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:50.481450 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:29:50.481763 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:50.482913 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:50.484356 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:50.485026 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:50.485167 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:50.485620 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:29:50.485775 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:29:50.485897 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:50.486051 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 20:29:50.486150 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 20:29:50.489017 5294 raft_consensus.cc:482] T 9ae589f6e03f4d5abf8db0d7c039927a P c03e7c30b2ef4546aed0e56cc6f3c685 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:29:50.489257 5294 raft_consensus.cc:504] T 9ae589f6e03f4d5abf8db0d7c039927a P c03e7c30b2ef4546aed0e56cc6f3c685 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:50.490731 5075 raft_consensus.cc:1261] T 9ae589f6e03f4d5abf8db0d7c039927a P d7103c3c79fe47f593e3c15745dd7ed7 [term 1 FOLLOWER]: Refusing update from remote peer 3c868738424b42cc824b67b80bd5b861: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:29:50.490821 4941 raft_consensus.cc:3037] T 9ae589f6e03f4d5abf8db0d7c039927a P c03e7c30b2ef4546aed0e56cc6f3c685 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:50.490983 5294 leader_election.cc:288] T 9ae589f6e03f4d5abf8db0d7c039927a P c03e7c30b2ef4546aed0e56cc6f3c685 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7103c3c79fe47f593e3c15745dd7ed7 (127.28.244.130:45127), 3c868738424b42cc824b67b80bd5b861 (127.28.244.131:44831)
I20260929 20:29:50.493160 5306 consensus_queue.cc:1035] T 9ae589f6e03f4d5abf8db0d7c039927a P 3c868738424b42cc824b67b80bd5b861 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:29:50.494611 4941 raft_consensus.cc:1261] T 9ae589f6e03f4d5abf8db0d7c039927a P c03e7c30b2ef4546aed0e56cc6f3c685 [term 1 FOLLOWER]: Refusing update from remote peer 3c868738424b42cc824b67b80bd5b861: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:29:50.496133 5306 consensus_queue.cc:1035] T 9ae589f6e03f4d5abf8db0d7c039927a P 3c868738424b42cc824b67b80bd5b861 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:29:50.497771 5296 tablet_replica.cc:439] T 9ae589f6e03f4d5abf8db0d7c039927a P d7103c3c79fe47f593e3c15745dd7ed7: TxnStatusTablet state changed. Reason: New leader 3c868738424b42cc824b67b80bd5b861. Latest consensus state: current_term: 1 leader_uuid: "3c868738424b42cc824b67b80bd5b861" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } } }
I20260929 20:29:50.498075 5296 tablet_replica.cc:442] T 9ae589f6e03f4d5abf8db0d7c039927a P d7103c3c79fe47f593e3c15745dd7ed7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:29:50.499588 5306 tablet_replica.cc:439] T 9ae589f6e03f4d5abf8db0d7c039927a P 3c868738424b42cc824b67b80bd5b861: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3c868738424b42cc824b67b80bd5b861" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } } }
I20260929 20:29:50.500036 5301 mvcc.cc:205] Tried to move back new op lower bound from 7334763685842276352 to 7334763685731057664. Current Snapshot: MvccSnapshot[applied={T|T < 7334763685842276352}]
I20260929 20:29:50.501618 5294 tablet_replica.cc:439] T 9ae589f6e03f4d5abf8db0d7c039927a P c03e7c30b2ef4546aed0e56cc6f3c685: TxnStatusTablet state changed. Reason: New leader 3c868738424b42cc824b67b80bd5b861. Latest consensus state: current_term: 1 leader_uuid: "3c868738424b42cc824b67b80bd5b861" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } } }
I20260929 20:29:50.501924 5294 tablet_replica.cc:442] T 9ae589f6e03f4d5abf8db0d7c039927a P c03e7c30b2ef4546aed0e56cc6f3c685: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:29:50.503468 5295 tablet_replica.cc:439] T 9ae589f6e03f4d5abf8db0d7c039927a P 3c868738424b42cc824b67b80bd5b861: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3c868738424b42cc824b67b80bd5b861" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } } }
I20260929 20:29:50.503705 5295 tablet_replica.cc:442] T 9ae589f6e03f4d5abf8db0d7c039927a P 3c868738424b42cc824b67b80bd5b861: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:29:50.504793 5302 mvcc.cc:205] Tried to move back new op lower bound from 7334763685842276352 to 7334763685731057664. Current Snapshot: MvccSnapshot[applied={T|T < 7334763685842276352}]
I20260929 20:29:50.505059 5296 tablet_replica.cc:439] T 9ae589f6e03f4d5abf8db0d7c039927a P d7103c3c79fe47f593e3c15745dd7ed7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3c868738424b42cc824b67b80bd5b861" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } } }
I20260929 20:29:50.505324 5296 tablet_replica.cc:442] T 9ae589f6e03f4d5abf8db0d7c039927a P d7103c3c79fe47f593e3c15745dd7ed7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:29:50.506225 5294 tablet_replica.cc:439] T 9ae589f6e03f4d5abf8db0d7c039927a P c03e7c30b2ef4546aed0e56cc6f3c685: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3c868738424b42cc824b67b80bd5b861" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } } }
I20260929 20:29:50.506488 5294 tablet_replica.cc:442] T 9ae589f6e03f4d5abf8db0d7c039927a P c03e7c30b2ef4546aed0e56cc6f3c685: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:29:50.510782 5306 tablet_replica.cc:442] T 9ae589f6e03f4d5abf8db0d7c039927a P 3c868738424b42cc824b67b80bd5b861: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:29:50.514822 5334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:50.491168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:45127 (local address 127.28.244.129:35183)
0929 20:29:50.491720 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:50.491739 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:50.491904 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:50.508312 (+ 16408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:50.508317 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:50.508342 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:29:50.508740 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:50.508753 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:50.510245 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:50.510250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:50.512262 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:50.512277 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:50.513996 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:50.514002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:50.514178 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:50.514209 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 20:29:50.514286 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 20:29:50.514446 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":472,"threads_started":1}
I20260929 20:29:50.515095 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:50.495460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:42125 (local address 127.28.244.131:44831)
0929 20:29:50.495619 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:50.495625 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:50.495729 (+ 104us) server_negotiation.cc:407] Connection header received
0929 20:29:50.496044 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:50.496049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:50.496121 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:50.496178 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:29:50.496874 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:50.498019 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:50.512610 (+ 14591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:50.513926 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:50.514053 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:50.514132 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:50.514673 (+ 541us) server_negotiation.cc:299] Negotiation successful
0929 20:29:50.514812 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 20:29:50.515702 5336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:50.495014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:44831 (local address 127.28.244.129:42125)
0929 20:29:50.495634 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:50.495656 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:50.495855 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:50.496357 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:50.496361 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:50.496385 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:29:50.496704 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:50.496717 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:50.498189 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:50.498194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:50.512421 (+ 14227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:50.512436 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:50.514327 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:50.514333 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:50.514487 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:50.514515 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 20:29:50.515238 (+ 723us) client_negotiation.cc:240] Negotiation successful
0929 20:29:50.515391 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":297,"threads_started":1}
I20260929 20:29:50.515959 5335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:50.491513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:35183 (local address 127.28.244.130:45127)
0929 20:29:50.507979 (+ 16466us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:50.507985 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:50.508006 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:29:50.508067 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:50.508072 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:50.508149 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:50.508908 (+ 759us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:29:50.509059 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:50.510102 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:50.512506 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:50.513735 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:50.515314 (+ 1579us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:50.515395 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:50.515443 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 20:29:50.515582 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16344,"thread_start_us":19750,"threads_started":1}
I20260929 20:29:50.516402 5075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ae589f6e03f4d5abf8db0d7c039927a" candidate_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" is_pre_election: true
I20260929 20:29:50.516353 5219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ae589f6e03f4d5abf8db0d7c039927a" candidate_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c868738424b42cc824b67b80bd5b861" is_pre_election: true
I20260929 20:29:50.517341 4869 leader_election.cc:302] T 9ae589f6e03f4d5abf8db0d7c039927a P c03e7c30b2ef4546aed0e56cc6f3c685 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c03e7c30b2ef4546aed0e56cc6f3c685; no voters: 3c868738424b42cc824b67b80bd5b861, d7103c3c79fe47f593e3c15745dd7ed7
I20260929 20:29:50.517819 5294 raft_consensus.cc:2726] T 9ae589f6e03f4d5abf8db0d7c039927a P c03e7c30b2ef4546aed0e56cc6f3c685 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 20:29:50.531131 5335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:50.512181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59096 (local address 127.28.244.130:45127)
0929 20:29:50.517249 (+ 5068us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:50.517255 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:50.524846 (+ 7591us) server_negotiation.cc:407] Connection header received
0929 20:29:50.525121 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:50.525125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:50.525207 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:50.525397 (+ 190us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:29:50.525943 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:50.526766 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:50.528851 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:50.529245 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:50.529342 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:50.529869 (+ 527us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:29:50.530496 (+ 627us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:29:50.530599 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:50.530744 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 20:29:50.530813 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4964,"thread_start_us":11873,"threads_started":1}
I20260929 20:29:50.549124 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:50.542154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:50155 (local address 127.28.244.131:44831)
0929 20:29:50.542387 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:50.542406 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:50.542918 (+ 512us) server_negotiation.cc:407] Connection header received
0929 20:29:50.543127 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:50.543131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:50.543199 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:50.543290 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:29:50.543965 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:50.544861 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:50.546987 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:50.547618 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:50.547728 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:50.548401 (+ 673us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:29:50.548544 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:29:50.548636 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:50.548808 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 20:29:50.548897 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:29:50.549422 5344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:50.542147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:44831 (local address 127.28.244.130:50155)
0929 20:29:50.542794 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:50.542814 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:50.542974 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:50.543320 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:50.543325 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:50.543339 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:29:50.543696 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:50.543710 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:50.545022 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:50.545028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:50.546781 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:50.546798 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:50.547831 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:50.547838 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:50.548035 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:50.548115 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:29:50.548661 (+ 546us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:29:50.548674 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 20:29:50.548767 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 20:29:50.548920 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":339,"threads_started":1}
I20260929 20:29:50.559443 5219 raft_consensus.cc:1261] T 1e889ba71a3d405a9bf9695f3a767947 P 3c868738424b42cc824b67b80bd5b861 [term 1 FOLLOWER]: Refusing update from remote peer d7103c3c79fe47f593e3c15745dd7ed7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:29:50.559495 4940 raft_consensus.cc:1261] T 1e889ba71a3d405a9bf9695f3a767947 P c03e7c30b2ef4546aed0e56cc6f3c685 [term 1 FOLLOWER]: Refusing update from remote peer d7103c3c79fe47f593e3c15745dd7ed7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:29:50.560202 5319 consensus_queue.cc:1035] T 1e889ba71a3d405a9bf9695f3a767947 P d7103c3c79fe47f593e3c15745dd7ed7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:29:50.560592 5296 consensus_queue.cc:1035] T 1e889ba71a3d405a9bf9695f3a767947 P d7103c3c79fe47f593e3c15745dd7ed7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:29:50.608366 5335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:50.601578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:57761 (local address 127.28.244.130:45127)
0929 20:29:50.601773 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:50.601782 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:50.602159 (+ 377us) server_negotiation.cc:407] Connection header received
0929 20:29:50.602374 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:50.602379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:50.602472 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:50.602576 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:29:50.603498 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:50.604719 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:50.606325 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:50.606976 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:50.607102 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:50.607585 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:29:50.607699 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:29:50.607799 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:50.607969 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 20:29:50.608052 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 20:29:50.608527 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:50.601318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:45127 (local address 127.28.244.131:57761)
0929 20:29:50.602058 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:50.602074 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:50.602237 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:50.602893 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:50.602898 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:50.602908 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:29:50.603321 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:50.603336 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:50.604882 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:50.604888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:50.606196 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:50.606211 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:50.607140 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:50.607144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:50.607282 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:50.607341 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:29:50.607799 (+ 458us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:29:50.607836 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 20:29:50.607945 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 20:29:50.608091 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":465,"threads_started":1}
I20260929 20:29:50.656250 4777 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:29:50.656574 4777 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:29:50.660526 4777 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 535a1a0ea12c439ab74e69d63b55b5ea: Sending DeleteTablet for 3 replicas of tablet 1e889ba71a3d405a9bf9695f3a767947
I20260929 20:29:50.661720 5055 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e889ba71a3d405a9bf9695f3a767947 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:29:50 UTC) from {username='oryx', principal='oryx/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:59090
I20260929 20:29:50.661824 4916 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e889ba71a3d405a9bf9695f3a767947 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:29:50 UTC) from {username='oryx', principal='oryx/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:34000
I20260929 20:29:50.662643 5194 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e889ba71a3d405a9bf9695f3a767947 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:29:50 UTC) from {username='oryx', principal='oryx/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:55622
I20260929 20:29:50.670496 5354 tablet_replica.cc:330] T 1e889ba71a3d405a9bf9695f3a767947 P 3c868738424b42cc824b67b80bd5b861: stopping tablet replica
I20260929 20:29:50.670732 5354 raft_consensus.cc:2229] T 1e889ba71a3d405a9bf9695f3a767947 P 3c868738424b42cc824b67b80bd5b861 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:29:50.671027 5354 raft_consensus.cc:2258] T 1e889ba71a3d405a9bf9695f3a767947 P 3c868738424b42cc824b67b80bd5b861 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:29:50.672281 5354 ts_tablet_manager.cc:1905] T 1e889ba71a3d405a9bf9695f3a767947 P 3c868738424b42cc824b67b80bd5b861: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:29:50.682649 5354 ts_tablet_manager.cc:1918] T 1e889ba71a3d405a9bf9695f3a767947 P 3c868738424b42cc824b67b80bd5b861: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:29:50.686962 5353 tablet_replica.cc:330] T 1e889ba71a3d405a9bf9695f3a767947 P c03e7c30b2ef4546aed0e56cc6f3c685: stopping tablet replica
I20260929 20:29:50.687461 5353 raft_consensus.cc:2229] T 1e889ba71a3d405a9bf9695f3a767947 P c03e7c30b2ef4546aed0e56cc6f3c685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:29:50.687819 5353 raft_consensus.cc:2258] T 1e889ba71a3d405a9bf9695f3a767947 P c03e7c30b2ef4546aed0e56cc6f3c685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:29:50.690505 5353 ts_tablet_manager.cc:1905] T 1e889ba71a3d405a9bf9695f3a767947 P c03e7c30b2ef4546aed0e56cc6f3c685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:29:50.693048 5352 tablet_replica.cc:330] T 1e889ba71a3d405a9bf9695f3a767947 P d7103c3c79fe47f593e3c15745dd7ed7: stopping tablet replica
I20260929 20:29:50.693437 5352 raft_consensus.cc:2229] T 1e889ba71a3d405a9bf9695f3a767947 P d7103c3c79fe47f593e3c15745dd7ed7 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:29:50.694144 5352 raft_consensus.cc:2258] T 1e889ba71a3d405a9bf9695f3a767947 P d7103c3c79fe47f593e3c15745dd7ed7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:29:50.694408 5354 log.cc:1196] T 1e889ba71a3d405a9bf9695f3a767947 P 3c868738424b42cc824b67b80bd5b861: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-2/wal/wals/1e889ba71a3d405a9bf9695f3a767947
I20260929 20:29:50.696264 5354 ts_tablet_manager.cc:1939] T 1e889ba71a3d405a9bf9695f3a767947 P 3c868738424b42cc824b67b80bd5b861: Deleting consensus metadata
I20260929 20:29:50.697703 4766 catalog_manager.cc:4841] TS 3c868738424b42cc824b67b80bd5b861 (127.28.244.131:44831): tablet 1e889ba71a3d405a9bf9695f3a767947 (table test-table [id=61a58d13fd424ce79c7ccbe778267b0c]) successfully deleted
I20260929 20:29:50.698554 5352 ts_tablet_manager.cc:1905] T 1e889ba71a3d405a9bf9695f3a767947 P d7103c3c79fe47f593e3c15745dd7ed7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:29:50.718713 5353 ts_tablet_manager.cc:1918] T 1e889ba71a3d405a9bf9695f3a767947 P c03e7c30b2ef4546aed0e56cc6f3c685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:29:50.718881 5353 log.cc:1196] T 1e889ba71a3d405a9bf9695f3a767947 P c03e7c30b2ef4546aed0e56cc6f3c685: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-0/wal/wals/1e889ba71a3d405a9bf9695f3a767947
I20260929 20:29:50.719337 5353 ts_tablet_manager.cc:1939] T 1e889ba71a3d405a9bf9695f3a767947 P c03e7c30b2ef4546aed0e56cc6f3c685: Deleting consensus metadata
I20260929 20:29:50.721192 4764 catalog_manager.cc:4841] TS c03e7c30b2ef4546aed0e56cc6f3c685 (127.28.244.129:45851): tablet 1e889ba71a3d405a9bf9695f3a767947 (table test-table [id=61a58d13fd424ce79c7ccbe778267b0c]) successfully deleted
I20260929 20:29:50.725131 5352 ts_tablet_manager.cc:1918] T 1e889ba71a3d405a9bf9695f3a767947 P d7103c3c79fe47f593e3c15745dd7ed7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:29:50.725386 5352 log.cc:1196] T 1e889ba71a3d405a9bf9695f3a767947 P d7103c3c79fe47f593e3c15745dd7ed7: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-1/wal/wals/1e889ba71a3d405a9bf9695f3a767947
I20260929 20:29:50.726032 5352 ts_tablet_manager.cc:1939] T 1e889ba71a3d405a9bf9695f3a767947 P d7103c3c79fe47f593e3c15745dd7ed7: Deleting consensus metadata
I20260929 20:29:50.727375 4765 catalog_manager.cc:4841] TS d7103c3c79fe47f593e3c15745dd7ed7 (127.28.244.130:45127): tablet 1e889ba71a3d405a9bf9695f3a767947 (table test-table [id=61a58d13fd424ce79c7ccbe778267b0c]) successfully deleted
Sep 29 20:29:50 dist-test-slave-p29l krb5kdc[4734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 20:29:50 dist-test-slave-p29l krb5kdc[4734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:29:50 dist-test-slave-p29l krb5kdc[4734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 20:29:50.786903 5327 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:29:50.773344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47594 (local address 127.28.244.190:35517)
0929 20:29:50.773574 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:50.773584 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:50.774028 (+ 444us) server_negotiation.cc:407] Connection header received
0929 20:29:50.774236 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:50.774244 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:50.774332 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:50.774464 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:50.776332 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:50.777574 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:50.779468 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:50.780178 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:50.780317 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:50.786616 (+ 6299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47594: BlockingRecv error: recv got EOF from 127.0.0.1:47594 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 20:29:50 dist-test-slave-p29l krb5kdc[4734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM
I20260929 20:29:50.813539 5327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:50.790630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47598 (local address 127.28.244.190:35517)
0929 20:29:50.790811 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:50.790820 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:50.795142 (+ 4322us) server_negotiation.cc:407] Connection header received
0929 20:29:50.795229 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:50.795235 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:50.795325 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:50.795533 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:50.797257 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:50.798421 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:50.800485 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:50.801187 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:50.801287 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:50.806716 (+ 5429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:50.806723 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:50.806728 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:50.806776 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:50.810301 (+ 3525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:50.811210 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:50.811216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:50.811220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:50.811373 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:50.811844 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:50.811850 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:50.811855 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:50.812239 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:50.812514 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:50.812856 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 20:29:50.813107 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:29:50.817821 4777 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47598:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:29:50.818567 4777 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:29:50.826339 5055 tablet_service.cc:1467] Processing CreateTablet for tablet 8a422b4713d44c5f846144e4971a8b67 (DEFAULT_TABLE table=test-table [id=03b59475101646358e34956fb42bb401]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:29:50.826403 4916 tablet_service.cc:1467] Processing CreateTablet for tablet 8a422b4713d44c5f846144e4971a8b67 (DEFAULT_TABLE table=test-table [id=03b59475101646358e34956fb42bb401]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:29:50.826817 4916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a422b4713d44c5f846144e4971a8b67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:50.826826 5055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a422b4713d44c5f846144e4971a8b67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:50.834245 5289 tablet_bootstrap.cc:492] T 8a422b4713d44c5f846144e4971a8b67 P d7103c3c79fe47f593e3c15745dd7ed7: Bootstrap starting.
I20260929 20:29:50.834239 5194 tablet_service.cc:1467] Processing CreateTablet for tablet 8a422b4713d44c5f846144e4971a8b67 (DEFAULT_TABLE table=test-table [id=03b59475101646358e34956fb42bb401]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:29:50.834726 5194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a422b4713d44c5f846144e4971a8b67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:50.836002 5289 tablet_bootstrap.cc:654] T 8a422b4713d44c5f846144e4971a8b67 P d7103c3c79fe47f593e3c15745dd7ed7: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:50.845397 5288 tablet_bootstrap.cc:492] T 8a422b4713d44c5f846144e4971a8b67 P c03e7c30b2ef4546aed0e56cc6f3c685: Bootstrap starting.
I20260929 20:29:50.846197 5289 tablet_bootstrap.cc:492] T 8a422b4713d44c5f846144e4971a8b67 P d7103c3c79fe47f593e3c15745dd7ed7: No bootstrap required, opened a new log
I20260929 20:29:50.846483 5289 ts_tablet_manager.cc:1397] T 8a422b4713d44c5f846144e4971a8b67 P d7103c3c79fe47f593e3c15745dd7ed7: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260929 20:29:50.847301 5288 tablet_bootstrap.cc:654] T 8a422b4713d44c5f846144e4971a8b67 P c03e7c30b2ef4546aed0e56cc6f3c685: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:50.847494 5289 raft_consensus.cc:348] T 8a422b4713d44c5f846144e4971a8b67 P d7103c3c79fe47f593e3c15745dd7ed7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:50.847919 5289 raft_consensus.cc:374] T 8a422b4713d44c5f846144e4971a8b67 P d7103c3c79fe47f593e3c15745dd7ed7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:50.848070 5289 raft_consensus.cc:729] T 8a422b4713d44c5f846144e4971a8b67 P d7103c3c79fe47f593e3c15745dd7ed7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7103c3c79fe47f593e3c15745dd7ed7, State: Initialized, Role: FOLLOWER
I20260929 20:29:50.848415 5289 consensus_queue.cc:260] T 8a422b4713d44c5f846144e4971a8b67 P d7103c3c79fe47f593e3c15745dd7ed7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:50.849843 5289 ts_tablet_manager.cc:1428] T 8a422b4713d44c5f846144e4971a8b67 P d7103c3c79fe47f593e3c15745dd7ed7: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260929 20:29:50.855412 5290 tablet_bootstrap.cc:492] T 8a422b4713d44c5f846144e4971a8b67 P 3c868738424b42cc824b67b80bd5b861: Bootstrap starting.
I20260929 20:29:50.857300 5290 tablet_bootstrap.cc:654] T 8a422b4713d44c5f846144e4971a8b67 P 3c868738424b42cc824b67b80bd5b861: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:50.861764 5288 tablet_bootstrap.cc:492] T 8a422b4713d44c5f846144e4971a8b67 P c03e7c30b2ef4546aed0e56cc6f3c685: No bootstrap required, opened a new log
I20260929 20:29:50.861928 5288 ts_tablet_manager.cc:1397] T 8a422b4713d44c5f846144e4971a8b67 P c03e7c30b2ef4546aed0e56cc6f3c685: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.004s
I20260929 20:29:50.862735 5288 raft_consensus.cc:348] T 8a422b4713d44c5f846144e4971a8b67 P c03e7c30b2ef4546aed0e56cc6f3c685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:50.863006 5288 raft_consensus.cc:374] T 8a422b4713d44c5f846144e4971a8b67 P c03e7c30b2ef4546aed0e56cc6f3c685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:50.863103 5288 raft_consensus.cc:729] T 8a422b4713d44c5f846144e4971a8b67 P c03e7c30b2ef4546aed0e56cc6f3c685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c03e7c30b2ef4546aed0e56cc6f3c685, State: Initialized, Role: FOLLOWER
I20260929 20:29:50.863379 5288 consensus_queue.cc:260] T 8a422b4713d44c5f846144e4971a8b67 P c03e7c30b2ef4546aed0e56cc6f3c685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:50.866328 5288 ts_tablet_manager.cc:1428] T 8a422b4713d44c5f846144e4971a8b67 P c03e7c30b2ef4546aed0e56cc6f3c685: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260929 20:29:50.870265 5290 tablet_bootstrap.cc:492] T 8a422b4713d44c5f846144e4971a8b67 P 3c868738424b42cc824b67b80bd5b861: No bootstrap required, opened a new log
I20260929 20:29:50.870450 5290 ts_tablet_manager.cc:1397] T 8a422b4713d44c5f846144e4971a8b67 P 3c868738424b42cc824b67b80bd5b861: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.006s
I20260929 20:29:50.871340 5290 raft_consensus.cc:348] T 8a422b4713d44c5f846144e4971a8b67 P 3c868738424b42cc824b67b80bd5b861 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:50.871646 5290 raft_consensus.cc:374] T 8a422b4713d44c5f846144e4971a8b67 P 3c868738424b42cc824b67b80bd5b861 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:50.871750 5290 raft_consensus.cc:729] T 8a422b4713d44c5f846144e4971a8b67 P 3c868738424b42cc824b67b80bd5b861 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c868738424b42cc824b67b80bd5b861, State: Initialized, Role: FOLLOWER
I20260929 20:29:50.872037 5290 consensus_queue.cc:260] T 8a422b4713d44c5f846144e4971a8b67 P 3c868738424b42cc824b67b80bd5b861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:50.872707 5290 ts_tablet_manager.cc:1428] T 8a422b4713d44c5f846144e4971a8b67 P 3c868738424b42cc824b67b80bd5b861: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 20:29:51.158154 5298 raft_consensus.cc:482] T 8a422b4713d44c5f846144e4971a8b67 P 3c868738424b42cc824b67b80bd5b861 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:29:51.158380 5298 raft_consensus.cc:504] T 8a422b4713d44c5f846144e4971a8b67 P 3c868738424b42cc824b67b80bd5b861 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:51.159102 5298 leader_election.cc:288] T 8a422b4713d44c5f846144e4971a8b67 P 3c868738424b42cc824b67b80bd5b861 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c03e7c30b2ef4546aed0e56cc6f3c685 (127.28.244.129:45851), d7103c3c79fe47f593e3c15745dd7ed7 (127.28.244.130:45127)
I20260929 20:29:51.159706 4941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a422b4713d44c5f846144e4971a8b67" candidate_uuid: "3c868738424b42cc824b67b80bd5b861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" is_pre_election: true
I20260929 20:29:51.159752 5075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a422b4713d44c5f846144e4971a8b67" candidate_uuid: "3c868738424b42cc824b67b80bd5b861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" is_pre_election: true
I20260929 20:29:51.160054 4941 raft_consensus.cc:2454] T 8a422b4713d44c5f846144e4971a8b67 P c03e7c30b2ef4546aed0e56cc6f3c685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c868738424b42cc824b67b80bd5b861 in term 0.
I20260929 20:29:51.160117 5075 raft_consensus.cc:2454] T 8a422b4713d44c5f846144e4971a8b67 P d7103c3c79fe47f593e3c15745dd7ed7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c868738424b42cc824b67b80bd5b861 in term 0.
I20260929 20:29:51.160682 5146 leader_election.cc:302] T 8a422b4713d44c5f846144e4971a8b67 P 3c868738424b42cc824b67b80bd5b861 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c868738424b42cc824b67b80bd5b861, c03e7c30b2ef4546aed0e56cc6f3c685; no voters:
I20260929 20:29:51.160990 5298 raft_consensus.cc:2781] T 8a422b4713d44c5f846144e4971a8b67 P 3c868738424b42cc824b67b80bd5b861 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:29:51.161099 5298 raft_consensus.cc:482] T 8a422b4713d44c5f846144e4971a8b67 P 3c868738424b42cc824b67b80bd5b861 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:29:51.161226 5298 raft_consensus.cc:3037] T 8a422b4713d44c5f846144e4971a8b67 P 3c868738424b42cc824b67b80bd5b861 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:51.163194 5298 raft_consensus.cc:504] T 8a422b4713d44c5f846144e4971a8b67 P 3c868738424b42cc824b67b80bd5b861 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:51.163861 5298 leader_election.cc:288] T 8a422b4713d44c5f846144e4971a8b67 P 3c868738424b42cc824b67b80bd5b861 [CANDIDATE]: Term 1 election: Requested vote from peers c03e7c30b2ef4546aed0e56cc6f3c685 (127.28.244.129:45851), d7103c3c79fe47f593e3c15745dd7ed7 (127.28.244.130:45127)
I20260929 20:29:51.164320 4941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a422b4713d44c5f846144e4971a8b67" candidate_uuid: "3c868738424b42cc824b67b80bd5b861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685"
I20260929 20:29:51.164397 5075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a422b4713d44c5f846144e4971a8b67" candidate_uuid: "3c868738424b42cc824b67b80bd5b861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7103c3c79fe47f593e3c15745dd7ed7"
I20260929 20:29:51.164597 4941 raft_consensus.cc:3037] T 8a422b4713d44c5f846144e4971a8b67 P c03e7c30b2ef4546aed0e56cc6f3c685 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:51.164628 5075 raft_consensus.cc:3037] T 8a422b4713d44c5f846144e4971a8b67 P d7103c3c79fe47f593e3c15745dd7ed7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:51.167692 5075 raft_consensus.cc:2454] T 8a422b4713d44c5f846144e4971a8b67 P d7103c3c79fe47f593e3c15745dd7ed7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c868738424b42cc824b67b80bd5b861 in term 1.
I20260929 20:29:51.167688 4941 raft_consensus.cc:2454] T 8a422b4713d44c5f846144e4971a8b67 P c03e7c30b2ef4546aed0e56cc6f3c685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c868738424b42cc824b67b80bd5b861 in term 1.
I20260929 20:29:51.168273 5146 leader_election.cc:302] T 8a422b4713d44c5f846144e4971a8b67 P 3c868738424b42cc824b67b80bd5b861 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3c868738424b42cc824b67b80bd5b861, c03e7c30b2ef4546aed0e56cc6f3c685, d7103c3c79fe47f593e3c15745dd7ed7; no voters:
I20260929 20:29:51.168638 5298 raft_consensus.cc:2781] T 8a422b4713d44c5f846144e4971a8b67 P 3c868738424b42cc824b67b80bd5b861 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:29:51.169015 5298 raft_consensus.cc:686] T 8a422b4713d44c5f846144e4971a8b67 P 3c868738424b42cc824b67b80bd5b861 [term 1 LEADER]: Becoming Leader. State: Replica: 3c868738424b42cc824b67b80bd5b861, State: Running, Role: LEADER
I20260929 20:29:51.169276 5298 consensus_queue.cc:237] T 8a422b4713d44c5f846144e4971a8b67 P 3c868738424b42cc824b67b80bd5b861 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:51.172889 4777 catalog_manager.cc:5495] T 8a422b4713d44c5f846144e4971a8b67 P 3c868738424b42cc824b67b80bd5b861 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c868738424b42cc824b67b80bd5b861 (127.28.244.131). New cstate: current_term: 1 leader_uuid: "3c868738424b42cc824b67b80bd5b861" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } health_report { overall_health: HEALTHY } } }
I20260929 20:29:51.223146 5373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:51.217609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55642 (local address 127.28.244.131:44831)
0929 20:29:51.218115 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:51.218120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:51.218136 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:29:51.218195 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:51.218199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:51.218260 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:51.218344 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:29:51.218774 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:51.219534 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:51.221000 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:51.221550 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:51.221655 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:51.222204 (+ 549us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:29:51.222320 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:29:51.222460 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:51.222688 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 20:29:51.222762 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":291,"threads_started":1}
I20260929 20:29:51.226475 4941 raft_consensus.cc:1261] T 8a422b4713d44c5f846144e4971a8b67 P c03e7c30b2ef4546aed0e56cc6f3c685 [term 1 FOLLOWER]: Refusing update from remote peer 3c868738424b42cc824b67b80bd5b861: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:29:51.226490 5075 raft_consensus.cc:1261] T 8a422b4713d44c5f846144e4971a8b67 P d7103c3c79fe47f593e3c15745dd7ed7 [term 1 FOLLOWER]: Refusing update from remote peer 3c868738424b42cc824b67b80bd5b861: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:29:51.227203 5370 consensus_queue.cc:1035] T 8a422b4713d44c5f846144e4971a8b67 P 3c868738424b42cc824b67b80bd5b861 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:29:51.227537 5298 consensus_queue.cc:1035] T 8a422b4713d44c5f846144e4971a8b67 P 3c868738424b42cc824b67b80bd5b861 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:29:51.241801 4778 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:29:51.242100 4778 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:29:51.245244 4778 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 535a1a0ea12c439ab74e69d63b55b5ea: Sending DeleteTablet for 3 replicas of tablet 8a422b4713d44c5f846144e4971a8b67
I20260929 20:29:51.246248 4916 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a422b4713d44c5f846144e4971a8b67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:29:51 UTC) from {username='oryx', principal='oryx/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:34000
I20260929 20:29:51.246328 5055 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a422b4713d44c5f846144e4971a8b67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:29:51 UTC) from {username='oryx', principal='oryx/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:59090
I20260929 20:29:51.246384 5194 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a422b4713d44c5f846144e4971a8b67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:29:51 UTC) from {username='oryx', principal='oryx/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:55622
I20260929 20:29:51.247014 5384 tablet_replica.cc:330] T 8a422b4713d44c5f846144e4971a8b67 P d7103c3c79fe47f593e3c15745dd7ed7: stopping tablet replica
I20260929 20:29:51.247231 5384 raft_consensus.cc:2229] T 8a422b4713d44c5f846144e4971a8b67 P d7103c3c79fe47f593e3c15745dd7ed7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:29:51.247300 5385 tablet_replica.cc:330] T 8a422b4713d44c5f846144e4971a8b67 P 3c868738424b42cc824b67b80bd5b861: stopping tablet replica
I20260929 20:29:51.247582 5384 raft_consensus.cc:2258] T 8a422b4713d44c5f846144e4971a8b67 P d7103c3c79fe47f593e3c15745dd7ed7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:29:51.247617 5385 raft_consensus.cc:2229] T 8a422b4713d44c5f846144e4971a8b67 P 3c868738424b42cc824b67b80bd5b861 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:29:51.248154 5385 raft_consensus.cc:2258] T 8a422b4713d44c5f846144e4971a8b67 P 3c868738424b42cc824b67b80bd5b861 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:29:51.248795 5384 ts_tablet_manager.cc:1905] T 8a422b4713d44c5f846144e4971a8b67 P d7103c3c79fe47f593e3c15745dd7ed7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:29:51.248584 4778 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47598:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:29:51.249519 4778 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:29:51.251478 5383 tablet_replica.cc:330] T 8a422b4713d44c5f846144e4971a8b67 P c03e7c30b2ef4546aed0e56cc6f3c685: stopping tablet replica
I20260929 20:29:51.251683 5383 raft_consensus.cc:2229] T 8a422b4713d44c5f846144e4971a8b67 P c03e7c30b2ef4546aed0e56cc6f3c685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:29:51.251989 5383 raft_consensus.cc:2258] T 8a422b4713d44c5f846144e4971a8b67 P c03e7c30b2ef4546aed0e56cc6f3c685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:29:51.252199 5385 ts_tablet_manager.cc:1905] T 8a422b4713d44c5f846144e4971a8b67 P 3c868738424b42cc824b67b80bd5b861: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:29:51.253161 5383 ts_tablet_manager.cc:1905] T 8a422b4713d44c5f846144e4971a8b67 P c03e7c30b2ef4546aed0e56cc6f3c685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:29:51.254631 5384 ts_tablet_manager.cc:1918] T 8a422b4713d44c5f846144e4971a8b67 P d7103c3c79fe47f593e3c15745dd7ed7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:29:51.254762 5384 log.cc:1196] T 8a422b4713d44c5f846144e4971a8b67 P d7103c3c79fe47f593e3c15745dd7ed7: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-1/wal/wals/8a422b4713d44c5f846144e4971a8b67
I20260929 20:29:51.255205 5384 ts_tablet_manager.cc:1939] T 8a422b4713d44c5f846144e4971a8b67 P d7103c3c79fe47f593e3c15745dd7ed7: Deleting consensus metadata
I20260929 20:29:51.256377 4765 catalog_manager.cc:4841] TS d7103c3c79fe47f593e3c15745dd7ed7 (127.28.244.130:45127): tablet 8a422b4713d44c5f846144e4971a8b67 (table test-table [id=03b59475101646358e34956fb42bb401]) successfully deleted
I20260929 20:29:51.259238 4916 tablet_service.cc:1467] Processing CreateTablet for tablet 7f28768317834381a604d5a727aea28e (DEFAULT_TABLE table=test-table [id=df266c94e09a46358aacdb844b05262d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:29:51.259670 4916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f28768317834381a604d5a727aea28e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:51.259634 5055 tablet_service.cc:1467] Processing CreateTablet for tablet 7f28768317834381a604d5a727aea28e (DEFAULT_TABLE table=test-table [id=df266c94e09a46358aacdb844b05262d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:29:51.260053 5055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f28768317834381a604d5a727aea28e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:51.260175 5383 ts_tablet_manager.cc:1918] T 8a422b4713d44c5f846144e4971a8b67 P c03e7c30b2ef4546aed0e56cc6f3c685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:29:51.260314 5383 log.cc:1196] T 8a422b4713d44c5f846144e4971a8b67 P c03e7c30b2ef4546aed0e56cc6f3c685: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-0/wal/wals/8a422b4713d44c5f846144e4971a8b67
I20260929 20:29:51.260769 5383 ts_tablet_manager.cc:1939] T 8a422b4713d44c5f846144e4971a8b67 P c03e7c30b2ef4546aed0e56cc6f3c685: Deleting consensus metadata
I20260929 20:29:51.262020 4764 catalog_manager.cc:4841] TS c03e7c30b2ef4546aed0e56cc6f3c685 (127.28.244.129:45851): tablet 8a422b4713d44c5f846144e4971a8b67 (table test-table [id=03b59475101646358e34956fb42bb401]) successfully deleted
I20260929 20:29:51.264732 5289 tablet_bootstrap.cc:492] T 7f28768317834381a604d5a727aea28e P d7103c3c79fe47f593e3c15745dd7ed7: Bootstrap starting.
I20260929 20:29:51.264881 5288 tablet_bootstrap.cc:492] T 7f28768317834381a604d5a727aea28e P c03e7c30b2ef4546aed0e56cc6f3c685: Bootstrap starting.
I20260929 20:29:51.265225 5194 tablet_service.cc:1467] Processing CreateTablet for tablet 7f28768317834381a604d5a727aea28e (DEFAULT_TABLE table=test-table [id=df266c94e09a46358aacdb844b05262d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:29:51.265672 5194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f28768317834381a604d5a727aea28e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:51.266426 5289 tablet_bootstrap.cc:654] T 7f28768317834381a604d5a727aea28e P d7103c3c79fe47f593e3c15745dd7ed7: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:51.266443 5288 tablet_bootstrap.cc:654] T 7f28768317834381a604d5a727aea28e P c03e7c30b2ef4546aed0e56cc6f3c685: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:51.266945 5385 ts_tablet_manager.cc:1918] T 8a422b4713d44c5f846144e4971a8b67 P 3c868738424b42cc824b67b80bd5b861: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:29:51.267169 5385 log.cc:1196] T 8a422b4713d44c5f846144e4971a8b67 P 3c868738424b42cc824b67b80bd5b861: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-2/wal/wals/8a422b4713d44c5f846144e4971a8b67
I20260929 20:29:51.267911 5385 ts_tablet_manager.cc:1939] T 8a422b4713d44c5f846144e4971a8b67 P 3c868738424b42cc824b67b80bd5b861: Deleting consensus metadata
I20260929 20:29:51.271112 4766 catalog_manager.cc:4841] TS 3c868738424b42cc824b67b80bd5b861 (127.28.244.131:44831): tablet 8a422b4713d44c5f846144e4971a8b67 (table test-table [id=03b59475101646358e34956fb42bb401]) successfully deleted
I20260929 20:29:51.272773 5288 tablet_bootstrap.cc:492] T 7f28768317834381a604d5a727aea28e P c03e7c30b2ef4546aed0e56cc6f3c685: No bootstrap required, opened a new log
I20260929 20:29:51.273098 5288 ts_tablet_manager.cc:1397] T 7f28768317834381a604d5a727aea28e P c03e7c30b2ef4546aed0e56cc6f3c685: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.001s
I20260929 20:29:51.273290 5289 tablet_bootstrap.cc:492] T 7f28768317834381a604d5a727aea28e P d7103c3c79fe47f593e3c15745dd7ed7: No bootstrap required, opened a new log
I20260929 20:29:51.273514 5289 ts_tablet_manager.cc:1397] T 7f28768317834381a604d5a727aea28e P d7103c3c79fe47f593e3c15745dd7ed7: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260929 20:29:51.273964 5288 raft_consensus.cc:348] T 7f28768317834381a604d5a727aea28e P c03e7c30b2ef4546aed0e56cc6f3c685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:51.274196 5290 tablet_bootstrap.cc:492] T 7f28768317834381a604d5a727aea28e P 3c868738424b42cc824b67b80bd5b861: Bootstrap starting.
I20260929 20:29:51.274240 5288 raft_consensus.cc:374] T 7f28768317834381a604d5a727aea28e P c03e7c30b2ef4546aed0e56cc6f3c685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:51.274222 5289 raft_consensus.cc:348] T 7f28768317834381a604d5a727aea28e P d7103c3c79fe47f593e3c15745dd7ed7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:51.274322 5288 raft_consensus.cc:729] T 7f28768317834381a604d5a727aea28e P c03e7c30b2ef4546aed0e56cc6f3c685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c03e7c30b2ef4546aed0e56cc6f3c685, State: Initialized, Role: FOLLOWER
I20260929 20:29:51.274454 5289 raft_consensus.cc:374] T 7f28768317834381a604d5a727aea28e P d7103c3c79fe47f593e3c15745dd7ed7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:51.274525 5289 raft_consensus.cc:729] T 7f28768317834381a604d5a727aea28e P d7103c3c79fe47f593e3c15745dd7ed7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7103c3c79fe47f593e3c15745dd7ed7, State: Initialized, Role: FOLLOWER
I20260929 20:29:51.274691 5288 consensus_queue.cc:260] T 7f28768317834381a604d5a727aea28e P c03e7c30b2ef4546aed0e56cc6f3c685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:51.274781 5289 consensus_queue.cc:260] T 7f28768317834381a604d5a727aea28e P d7103c3c79fe47f593e3c15745dd7ed7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:51.275323 5289 ts_tablet_manager.cc:1428] T 7f28768317834381a604d5a727aea28e P d7103c3c79fe47f593e3c15745dd7ed7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 20:29:51.276042 5290 tablet_bootstrap.cc:654] T 7f28768317834381a604d5a727aea28e P 3c868738424b42cc824b67b80bd5b861: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:51.276813 5288 ts_tablet_manager.cc:1428] T 7f28768317834381a604d5a727aea28e P c03e7c30b2ef4546aed0e56cc6f3c685: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260929 20:29:51.278169 5290 tablet_bootstrap.cc:492] T 7f28768317834381a604d5a727aea28e P 3c868738424b42cc824b67b80bd5b861: No bootstrap required, opened a new log
I20260929 20:29:51.278349 5290 ts_tablet_manager.cc:1397] T 7f28768317834381a604d5a727aea28e P 3c868738424b42cc824b67b80bd5b861: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260929 20:29:51.279016 5290 raft_consensus.cc:348] T 7f28768317834381a604d5a727aea28e P 3c868738424b42cc824b67b80bd5b861 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:51.279233 5290 raft_consensus.cc:374] T 7f28768317834381a604d5a727aea28e P 3c868738424b42cc824b67b80bd5b861 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:51.279313 5290 raft_consensus.cc:729] T 7f28768317834381a604d5a727aea28e P 3c868738424b42cc824b67b80bd5b861 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c868738424b42cc824b67b80bd5b861, State: Initialized, Role: FOLLOWER
I20260929 20:29:51.279587 5290 consensus_queue.cc:260] T 7f28768317834381a604d5a727aea28e P 3c868738424b42cc824b67b80bd5b861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:51.280190 5290 ts_tablet_manager.cc:1428] T 7f28768317834381a604d5a727aea28e P 3c868738424b42cc824b67b80bd5b861: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 20:29:51.281813 5298 raft_consensus.cc:482] T 7f28768317834381a604d5a727aea28e P 3c868738424b42cc824b67b80bd5b861 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:29:51.281985 5298 raft_consensus.cc:504] T 7f28768317834381a604d5a727aea28e P 3c868738424b42cc824b67b80bd5b861 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:51.282574 5298 leader_election.cc:288] T 7f28768317834381a604d5a727aea28e P 3c868738424b42cc824b67b80bd5b861 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c03e7c30b2ef4546aed0e56cc6f3c685 (127.28.244.129:45851), d7103c3c79fe47f593e3c15745dd7ed7 (127.28.244.130:45127)
I20260929 20:29:51.283015 4941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f28768317834381a604d5a727aea28e" candidate_uuid: "3c868738424b42cc824b67b80bd5b861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" is_pre_election: true
I20260929 20:29:51.283164 5075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f28768317834381a604d5a727aea28e" candidate_uuid: "3c868738424b42cc824b67b80bd5b861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" is_pre_election: true
I20260929 20:29:51.283277 4941 raft_consensus.cc:2454] T 7f28768317834381a604d5a727aea28e P c03e7c30b2ef4546aed0e56cc6f3c685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c868738424b42cc824b67b80bd5b861 in term 0.
I20260929 20:29:51.283511 5075 raft_consensus.cc:2454] T 7f28768317834381a604d5a727aea28e P d7103c3c79fe47f593e3c15745dd7ed7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c868738424b42cc824b67b80bd5b861 in term 0.
I20260929 20:29:51.283679 5146 leader_election.cc:302] T 7f28768317834381a604d5a727aea28e P 3c868738424b42cc824b67b80bd5b861 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c868738424b42cc824b67b80bd5b861, c03e7c30b2ef4546aed0e56cc6f3c685; no voters:
I20260929 20:29:51.284129 5298 raft_consensus.cc:2781] T 7f28768317834381a604d5a727aea28e P 3c868738424b42cc824b67b80bd5b861 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:29:51.284235 5298 raft_consensus.cc:482] T 7f28768317834381a604d5a727aea28e P 3c868738424b42cc824b67b80bd5b861 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:29:51.284340 5298 raft_consensus.cc:3037] T 7f28768317834381a604d5a727aea28e P 3c868738424b42cc824b67b80bd5b861 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:51.286729 5298 raft_consensus.cc:504] T 7f28768317834381a604d5a727aea28e P 3c868738424b42cc824b67b80bd5b861 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:51.287317 5298 leader_election.cc:288] T 7f28768317834381a604d5a727aea28e P 3c868738424b42cc824b67b80bd5b861 [CANDIDATE]: Term 1 election: Requested vote from peers c03e7c30b2ef4546aed0e56cc6f3c685 (127.28.244.129:45851), d7103c3c79fe47f593e3c15745dd7ed7 (127.28.244.130:45127)
I20260929 20:29:51.287817 4941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f28768317834381a604d5a727aea28e" candidate_uuid: "3c868738424b42cc824b67b80bd5b861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685"
I20260929 20:29:51.287925 5075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f28768317834381a604d5a727aea28e" candidate_uuid: "3c868738424b42cc824b67b80bd5b861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7103c3c79fe47f593e3c15745dd7ed7"
I20260929 20:29:51.288133 4941 raft_consensus.cc:3037] T 7f28768317834381a604d5a727aea28e P c03e7c30b2ef4546aed0e56cc6f3c685 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:51.288136 5075 raft_consensus.cc:3037] T 7f28768317834381a604d5a727aea28e P d7103c3c79fe47f593e3c15745dd7ed7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:51.291138 5075 raft_consensus.cc:2454] T 7f28768317834381a604d5a727aea28e P d7103c3c79fe47f593e3c15745dd7ed7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c868738424b42cc824b67b80bd5b861 in term 1.
I20260929 20:29:51.291147 4941 raft_consensus.cc:2454] T 7f28768317834381a604d5a727aea28e P c03e7c30b2ef4546aed0e56cc6f3c685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c868738424b42cc824b67b80bd5b861 in term 1.
I20260929 20:29:51.291579 5147 leader_election.cc:302] T 7f28768317834381a604d5a727aea28e P 3c868738424b42cc824b67b80bd5b861 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c868738424b42cc824b67b80bd5b861, d7103c3c79fe47f593e3c15745dd7ed7; no voters:
I20260929 20:29:51.291857 5298 raft_consensus.cc:2781] T 7f28768317834381a604d5a727aea28e P 3c868738424b42cc824b67b80bd5b861 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:29:51.292127 5298 raft_consensus.cc:686] T 7f28768317834381a604d5a727aea28e P 3c868738424b42cc824b67b80bd5b861 [term 1 LEADER]: Becoming Leader. State: Replica: 3c868738424b42cc824b67b80bd5b861, State: Running, Role: LEADER
I20260929 20:29:51.292423 5298 consensus_queue.cc:237] T 7f28768317834381a604d5a727aea28e P 3c868738424b42cc824b67b80bd5b861 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } }
I20260929 20:29:51.295508 4778 catalog_manager.cc:5495] T 7f28768317834381a604d5a727aea28e P 3c868738424b42cc824b67b80bd5b861 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c868738424b42cc824b67b80bd5b861 (127.28.244.131). New cstate: current_term: 1 leader_uuid: "3c868738424b42cc824b67b80bd5b861" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c868738424b42cc824b67b80bd5b861" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44831 } health_report { overall_health: HEALTHY } } }
I20260929 20:29:51.324774 5373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:51.319419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:46955 (local address 127.28.244.131:44831)
0929 20:29:51.319603 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:51.319607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:51.320048 (+ 441us) server_negotiation.cc:407] Connection header received
0929 20:29:51.320225 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:51.320231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:51.320303 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:51.320423 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:29:51.320932 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:51.321686 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:51.323003 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:51.323440 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:51.323538 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:51.323994 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:29:51.324102 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:29:51.324190 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:51.324326 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 20:29:51.324392 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":66}
I20260929 20:29:51.324869 5392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:51.319227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:44831 (local address 127.28.244.131:46955)
0929 20:29:51.319957 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:51.319973 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:51.320088 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:51.320480 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:51.320488 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:51.320504 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:29:51.320784 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:51.320792 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:51.321829 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:51.321833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:51.322875 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:51.322886 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:51.323588 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:51.323593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:51.323722 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:51.323760 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:29:51.324204 (+ 444us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:29:51.324215 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 20:29:51.324291 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 20:29:51.324513 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":38,"thread_start_us":430,"threads_started":1}
I20260929 20:29:51.333334 4941 raft_consensus.cc:1261] T 7f28768317834381a604d5a727aea28e P c03e7c30b2ef4546aed0e56cc6f3c685 [term 1 FOLLOWER]: Refusing update from remote peer 3c868738424b42cc824b67b80bd5b861: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:29:51.333334 5075 raft_consensus.cc:1261] T 7f28768317834381a604d5a727aea28e P d7103c3c79fe47f593e3c15745dd7ed7 [term 1 FOLLOWER]: Refusing update from remote peer 3c868738424b42cc824b67b80bd5b861: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:29:51.333956 5298 consensus_queue.cc:1035] T 7f28768317834381a604d5a727aea28e P 3c868738424b42cc824b67b80bd5b861 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7103c3c79fe47f593e3c15745dd7ed7" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:29:51.334268 5379 consensus_queue.cc:1035] T 7f28768317834381a604d5a727aea28e P 3c868738424b42cc824b67b80bd5b861 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c03e7c30b2ef4546aed0e56cc6f3c685" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 45851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:29:51.385474 4779 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:29:51.385686 4779 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:29:51.389048 4779 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 535a1a0ea12c439ab74e69d63b55b5ea: Sending DeleteTablet for 3 replicas of tablet 7f28768317834381a604d5a727aea28e
I20260929 20:29:51.389964 5055 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f28768317834381a604d5a727aea28e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:29:51 UTC) from {username='oryx', principal='oryx/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:59090
I20260929 20:29:51.390223 5193 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f28768317834381a604d5a727aea28e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:29:51 UTC) from {username='oryx', principal='oryx/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:55622
I20260929 20:29:51.390292 4916 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f28768317834381a604d5a727aea28e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:29:51 UTC) from {username='oryx', principal='oryx/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:34000
I20260929 20:29:51.390586 5384 tablet_replica.cc:330] T 7f28768317834381a604d5a727aea28e P d7103c3c79fe47f593e3c15745dd7ed7: stopping tablet replica
I20260929 20:29:51.390810 5384 raft_consensus.cc:2229] T 7f28768317834381a604d5a727aea28e P d7103c3c79fe47f593e3c15745dd7ed7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:29:51.391017 5384 raft_consensus.cc:2258] T 7f28768317834381a604d5a727aea28e P d7103c3c79fe47f593e3c15745dd7ed7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:29:51.391305 5385 tablet_replica.cc:330] T 7f28768317834381a604d5a727aea28e P 3c868738424b42cc824b67b80bd5b861: stopping tablet replica
I20260929 20:29:51.391527 5385 raft_consensus.cc:2229] T 7f28768317834381a604d5a727aea28e P 3c868738424b42cc824b67b80bd5b861 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:29:51.391911 5385 raft_consensus.cc:2258] T 7f28768317834381a604d5a727aea28e P 3c868738424b42cc824b67b80bd5b861 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:29:51.391970 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 4850
I20260929 20:29:51.392215 5383 tablet_replica.cc:330] T 7f28768317834381a604d5a727aea28e P c03e7c30b2ef4546aed0e56cc6f3c685: stopping tablet replica
I20260929 20:29:51.392441 5383 raft_consensus.cc:2229] T 7f28768317834381a604d5a727aea28e P c03e7c30b2ef4546aed0e56cc6f3c685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:29:51.392637 5383 raft_consensus.cc:2258] T 7f28768317834381a604d5a727aea28e P c03e7c30b2ef4546aed0e56cc6f3c685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:29:51.393195 5385 ts_tablet_manager.cc:1905] T 7f28768317834381a604d5a727aea28e P 3c868738424b42cc824b67b80bd5b861: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:29:51.393642 5383 ts_tablet_manager.cc:1905] T 7f28768317834381a604d5a727aea28e P c03e7c30b2ef4546aed0e56cc6f3c685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:29:51.395365 5384 ts_tablet_manager.cc:1905] T 7f28768317834381a604d5a727aea28e P d7103c3c79fe47f593e3c15745dd7ed7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:29:51.398399 5385 ts_tablet_manager.cc:1918] T 7f28768317834381a604d5a727aea28e P 3c868738424b42cc824b67b80bd5b861: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:29:51.398583 5385 log.cc:1196] T 7f28768317834381a604d5a727aea28e P 3c868738424b42cc824b67b80bd5b861: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-2/wal/wals/7f28768317834381a604d5a727aea28e
I20260929 20:29:51.398940 5385 ts_tablet_manager.cc:1939] T 7f28768317834381a604d5a727aea28e P 3c868738424b42cc824b67b80bd5b861: Deleting consensus metadata
I20260929 20:29:51.399969 4766 catalog_manager.cc:4841] TS 3c868738424b42cc824b67b80bd5b861 (127.28.244.131:44831): tablet 7f28768317834381a604d5a727aea28e (table test-table [id=df266c94e09a46358aacdb844b05262d]) successfully deleted
I20260929 20:29:51.400972 5383 ts_tablet_manager.cc:1918] T 7f28768317834381a604d5a727aea28e P c03e7c30b2ef4546aed0e56cc6f3c685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:29:51.401170 5383 log.cc:1196] T 7f28768317834381a604d5a727aea28e P c03e7c30b2ef4546aed0e56cc6f3c685: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-0/wal/wals/7f28768317834381a604d5a727aea28e
I20260929 20:29:51.401616 5383 ts_tablet_manager.cc:1939] T 7f28768317834381a604d5a727aea28e P c03e7c30b2ef4546aed0e56cc6f3c685: Deleting consensus metadata
I20260929 20:29:51.402872 4764 catalog_manager.cc:4841] TS c03e7c30b2ef4546aed0e56cc6f3c685 (127.28.244.129:45851): tablet 7f28768317834381a604d5a727aea28e (table test-table [id=df266c94e09a46358aacdb844b05262d]) successfully deleted
I20260929 20:29:51.403661 5384 ts_tablet_manager.cc:1918] T 7f28768317834381a604d5a727aea28e P d7103c3c79fe47f593e3c15745dd7ed7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:29:51.404139 5384 log.cc:1196] T 7f28768317834381a604d5a727aea28e P d7103c3c79fe47f593e3c15745dd7ed7: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713721201454-29650-0/minicluster-data/ts-1/wal/wals/7f28768317834381a604d5a727aea28e
I20260929 20:29:51.404603 5384 ts_tablet_manager.cc:1939] T 7f28768317834381a604d5a727aea28e P d7103c3c79fe47f593e3c15745dd7ed7: Deleting consensus metadata
I20260929 20:29:51.406073 4765 catalog_manager.cc:4841] TS d7103c3c79fe47f593e3c15745dd7ed7 (127.28.244.130:45127): tablet 7f28768317834381a604d5a727aea28e (table test-table [id=df266c94e09a46358aacdb844b05262d]) successfully deleted
Sep 29 20:29:51 dist-test-slave-p29l krb5kdc[4734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.244.129@KRBTEST.COM
I20260929 20:29:51.431857 5397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:51.393562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34008 (local address 127.28.244.129:45851)
0929 20:29:51.404249 (+ 10687us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:51.404257 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:51.404283 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:29:51.404358 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:51.404365 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:51.404459 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:51.404599 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:51.406496 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:51.407747 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:51.409102 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:51.409877 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:51.410011 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:51.425098 (+ 15087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:51.425366 (+ 268us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:51.425386 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:51.425439 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:51.428746 (+ 3307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:51.429728 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:51.429736 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:51.429739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:51.429871 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:51.430360 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:51.430363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:51.430365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:51.430715 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:51.430967 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:51.431343 (+ 376us) server_negotiation.cc:299] Negotiation successful
0929 20:29:51.431535 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10497,"thread_start_us":10432,"threads_started":1}
I20260929 20:29:51.432621 4981 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:29:51.553025 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 4850
I20260929 20:29:51.572281 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 4990
Sep 29 20:29:51 dist-test-slave-p29l krb5kdc[4734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.244.130@KRBTEST.COM
I20260929 20:29:51.587182 5399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:51.573161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59102 (local address 127.28.244.130:45127)
0929 20:29:51.573730 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:51.573742 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:51.573757 (+ 15us) server_negotiation.cc:407] Connection header received
0929 20:29:51.573814 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:51.573818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:51.573877 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:51.574005 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:51.575236 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:51.576021 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:51.577257 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:51.577660 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:51.577795 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:51.581857 (+ 4062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:51.581871 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:51.581889 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:51.581934 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:51.584703 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:51.585342 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:51.585349 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:51.585352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:51.585529 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:51.585827 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:51.585830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:51.585832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:51.586160 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:51.586383 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:51.586618 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 20:29:51.586786 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":258,"threads_started":1}
I20260929 20:29:51.588017 5120 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:29:51.714248 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 4990
I20260929 20:29:51.738272 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 5129
Sep 29 20:29:51 dist-test-slave-p29l krb5kdc[4734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.244.131@KRBTEST.COM
I20260929 20:29:51.754323 5373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:51.739128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55654 (local address 127.28.244.131:44831)
0929 20:29:51.739305 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:51.739310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:51.739327 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:29:51.739446 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:51.739449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:51.739510 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:51.739597 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:51.740991 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:51.741787 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:51.743207 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:51.743716 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:51.743902 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:51.748608 (+ 4706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:51.748625 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:51.748643 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:51.748691 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:51.751577 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:51.752430 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:51.752438 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:51.752440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:51.752620 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:51.753023 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:51.753027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:51.753029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:51.753369 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:51.753631 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:51.753868 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 20:29:51.754040 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:29:51.755107 5259 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:29:51.904148 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 5129
I20260929 20:29:51.930653 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 4747
I20260929 20:29:51.931795 4819 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:29:52.051517 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 4747
2026-09-29T20:29:52Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5882 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:29:52 dist-test-slave-p29l krb5kdc[5405](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:29:52 dist-test-slave-p29l krb5kdc[5405](info): set up 2 sockets
Sep 29 20:29:52 dist-test-slave-p29l krb5kdc[5405](info): commencing operation
krb5kdc: starting...
W20260929 20:29:53.176358 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:29:53 dist-test-slave-p29l krb5kdc[5405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:29:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:29:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.244.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
I20260929 20:29:53.427511 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:38429
--webserver_interface=127.28.244.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:38429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:29:53.768965 5418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:53.769251 5418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:53.769424 5418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:53.776823 5418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:29:53.776945 5418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:53.777045 5418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:53.777122 5418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:29:53.777189 5418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:29:53.787662 5418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:38429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:38429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:53.789140 5418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:53.790527 5418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:53.802863 5426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:53.802873 5424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:53.802822 5423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:53.804064 5418 server_base.cc:1034] running on GCE node
I20260929 20:29:53.805063 5418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:53.806668 5418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:53.807934 5418 hybrid_clock.cc:648] HybridClock initialized: now 1790713793807876 us; error 42 us; skew 500 ppm
I20260929 20:29:53.808298 5418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T20:29:53Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1822 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:29:53 dist-test-slave-p29l krb5kdc[5432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:29:53 dist-test-slave-p29l krb5kdc[5432](info): set up 2 sockets
Sep 29 20:29:53 dist-test-slave-p29l krb5kdc[5432](info): commencing operation
krb5kdc: starting...
W20260929 20:29:54.997200 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:29:55 dist-test-slave-p29l krb5kdc[5432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:29:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:29:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.244.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
I20260929 20:29:55.239602 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:45043
--webserver_interface=127.28.244.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:33671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:45043
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:29:55.581862 5445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:55.582218 5445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:55.582391 5445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:55.589756 5445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:29:55.589916 5445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:55.590019 5445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:55.590121 5445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:29:55.590197 5445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:29:55.600262 5445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:33671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:45043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:45043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:55.601684 5445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:55.603025 5445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:55.616293 5450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:55.616317 5451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:55.616334 5453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:55.617255 5445 server_base.cc:1034] running on GCE node
I20260929 20:29:55.618117 5445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:55.619630 5445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:55.620872 5445 hybrid_clock.cc:648] HybridClock initialized: now 1790713795620772 us; error 91 us; skew 500 ppm
I20260929 20:29:55.621268 5445 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:55 dist-test-slave-p29l krb5kdc[5432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713795, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:55.624495 5445 init.cc:377] Logged in from keytab as kudu/127.28.244.190@KRBTEST.COM (short username kudu)
I20260929 20:29:55.625999 5445 webserver.cc:460] Webserver started at http://127.28.244.190:33821/ using document root <none> and password file <none>
I20260929 20:29:55.626670 5445 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:55.626768 5445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:55.627036 5445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:55.630031 5445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "ca7f9fb997004ee4a4710604dcfc1e93"
format_stamp: "Formatted at 2026-09-29 20:29:55 on dist-test-slave-p29l"
server_key: "0f3d05225474e55eba507152e2ec5f0e"
server_key_iv: "6f15e62ff112aec2a62e0d2a671486ee"
server_key_version: "encryptionkey@0"
I20260929 20:29:55.630759 5445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "ca7f9fb997004ee4a4710604dcfc1e93"
format_stamp: "Formatted at 2026-09-29 20:29:55 on dist-test-slave-p29l"
server_key: "0f3d05225474e55eba507152e2ec5f0e"
server_key_iv: "6f15e62ff112aec2a62e0d2a671486ee"
server_key_version: "encryptionkey@0"
I20260929 20:29:55.636132 5445 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260929 20:29:55.639707 5460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:55.641176 5445 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:29:55.641369 5445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "ca7f9fb997004ee4a4710604dcfc1e93"
format_stamp: "Formatted at 2026-09-29 20:29:55 on dist-test-slave-p29l"
server_key: "0f3d05225474e55eba507152e2ec5f0e"
server_key_iv: "6f15e62ff112aec2a62e0d2a671486ee"
server_key_version: "encryptionkey@0"
I20260929 20:29:55.641579 5445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:55.672475 5445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:55.676990 5445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:55.677313 5445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:55.696971 5445 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:45043
I20260929 20:29:55.696991 5511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:45043 every 8 connection(s)
I20260929 20:29:55.698422 5445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:29:55.702237 5512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:29:55.703178 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 5445
I20260929 20:29:55.703393 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:29:55.704289 29650 external_mini_cluster.cc:1428] Setting key 25172f087e5ecf74907a5b78c8c67524
I20260929 20:29:55.717358 5512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93: Bootstrap starting.
Sep 29 20:29:55 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:29:55.721382 5512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93: Neither blocks nor log segments found. Creating new log.
I20260929 20:29:55.722856 5512 log.cc:824] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93: Log is configured to *not* fsync() on all Append() calls
I20260929 20:29:55.726392 5512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93: No bootstrap required, opened a new log
I20260929 20:29:55.729385 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:55.706446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35416 (local address 127.28.244.190:45043)
0929 20:29:55.707353 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:55.707365 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:55.707408 (+ 43us) server_negotiation.cc:407] Connection header received
0929 20:29:55.709130 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:55.709156 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:55.709653 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:55.710025 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:55.711884 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:55.713240 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:55.715055 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:55.715796 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:55.716011 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:55.720644 (+ 4633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:55.720654 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:55.720673 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:55.720727 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:55.724675 (+ 3948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:55.725455 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:55.725463 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:55.725472 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:55.725688 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:55.726059 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:55.726064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:55.726067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:55.726771 (+ 704us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:55.727099 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:55.727418 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 20:29:55.727891 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":431,"threads_started":1}
I20260929 20:29:55.735595 5512 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7f9fb997004ee4a4710604dcfc1e93" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 45043 } }
I20260929 20:29:55.736045 5512 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:29:55.736126 5512 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca7f9fb997004ee4a4710604dcfc1e93, State: Initialized, Role: FOLLOWER
I20260929 20:29:55.736631 5512 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7f9fb997004ee4a4710604dcfc1e93" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 45043 } }
I20260929 20:29:55.736860 5512 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:29:55.737005 5512 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:29:55.737169 5512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:29:55.739658 5512 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7f9fb997004ee4a4710604dcfc1e93" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 45043 } }
I20260929 20:29:55.740331 5512 leader_election.cc:302] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca7f9fb997004ee4a4710604dcfc1e93; no voters:
I20260929 20:29:55.740963 5512 leader_election.cc:288] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:29:55.741133 5517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:29:55.741931 5517 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93 [term 1 LEADER]: Becoming Leader. State: Replica: ca7f9fb997004ee4a4710604dcfc1e93, State: Running, Role: LEADER
I20260929 20:29:55.742564 5517 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7f9fb997004ee4a4710604dcfc1e93" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 45043 } }
I20260929 20:29:55.743166 5512 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:29:55.745600 5518 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca7f9fb997004ee4a4710604dcfc1e93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7f9fb997004ee4a4710604dcfc1e93" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 45043 } } }
I20260929 20:29:55.745980 5518 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93 [sys.catalog]: This master's current role is: LEADER
I20260929 20:29:55.746753 5519 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca7f9fb997004ee4a4710604dcfc1e93. Latest consensus state: current_term: 1 leader_uuid: "ca7f9fb997004ee4a4710604dcfc1e93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7f9fb997004ee4a4710604dcfc1e93" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 45043 } } }
I20260929 20:29:55.747031 5523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:29:55.747088 5519 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93 [sys.catalog]: This master's current role is: LEADER
I20260929 20:29:55.753085 5523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:29:55.762250 5523 catalog_manager.cc:1347] Generated new cluster ID: 0057c5066ef5422baab535de48ec7719
I20260929 20:29:55.762372 5523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:29:55.777565 5523 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:29:55.778893 5523 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:29:55.787159 5523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93: Generated new TSK 0
I20260929 20:29:55.788061 5523 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.244.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab.
Entry for principal kudu/127.28.244.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab.
I20260929 20:29:55.853259 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.129:0
--local_ip_for_outbound_sockets=127.28.244.129
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:45043
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:33671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:29:56.200352 5538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:56.200713 5538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:56.200893 5538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:56.208455 5538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:56.208645 5538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:56.208824 5538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.129
I20260929 20:29:56.219491 5538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:33671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:45043
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.129
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:56.220963 5538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:56.222342 5538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:56.236614 5544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:56.236779 5543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:56.236614 5546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:56.237126 5538 server_base.cc:1034] running on GCE node
I20260929 20:29:56.237933 5538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:56.239053 5538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:56.240283 5538 hybrid_clock.cc:648] HybridClock initialized: now 1790713796240209 us; error 58 us; skew 500 ppm
I20260929 20:29:56.240631 5538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:56 dist-test-slave-p29l krb5kdc[5432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713796, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:56.244175 5538 init.cc:377] Logged in from keytab as kudu/127.28.244.129@KRBTEST.COM (short username kudu)
I20260929 20:29:56.246076 5538 webserver.cc:460] Webserver started at http://127.28.244.129:34213/ using document root <none> and password file <none>
I20260929 20:29:56.246920 5538 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:56.247028 5538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:56.247290 5538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:56.250478 5538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-0/data/instance:
uuid: "016699d1812448c4bb4fa4a17b00983f"
format_stamp: "Formatted at 2026-09-29 20:29:56 on dist-test-slave-p29l"
server_key: "2f66a0cfad152050cbda4592f01f9ba1"
server_key_iv: "a314c84e469aa7b135ad9db50e886363"
server_key_version: "encryptionkey@0"
I20260929 20:29:56.251253 5538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance:
uuid: "016699d1812448c4bb4fa4a17b00983f"
format_stamp: "Formatted at 2026-09-29 20:29:56 on dist-test-slave-p29l"
server_key: "2f66a0cfad152050cbda4592f01f9ba1"
server_key_iv: "a314c84e469aa7b135ad9db50e886363"
server_key_version: "encryptionkey@0"
I20260929 20:29:56.257308 5538 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.006s
I20260929 20:29:56.261288 5553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:56.262784 5538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:29:56.262945 5538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-0/wal
uuid: "016699d1812448c4bb4fa4a17b00983f"
format_stamp: "Formatted at 2026-09-29 20:29:56 on dist-test-slave-p29l"
server_key: "2f66a0cfad152050cbda4592f01f9ba1"
server_key_iv: "a314c84e469aa7b135ad9db50e886363"
server_key_version: "encryptionkey@0"
I20260929 20:29:56.263125 5538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:56.280268 5538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:56.284905 5538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:56.285162 5538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:56.286238 5538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:29:56.287739 5538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:29:56.287868 5538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 20:29:56.288029 5538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:29:56.288118 5538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:56.323442 5538 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.129:44727
I20260929 20:29:56.323501 5665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.129:44727 every 8 connection(s)
I20260929 20:29:56.324879 5538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
I20260929 20:29:56.331242 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 5538
I20260929 20:29:56.331478 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance
I20260929 20:29:56.332197 29650 external_mini_cluster.cc:1428] Setting key 054c8ae5873f0a7ae1f06fb8da35b18b
Sep 29 20:29:56 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713796, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:29:56.348837 5668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:56.327679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:46533 (local address 127.28.244.190:45043)
0929 20:29:56.328224 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:56.328230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:56.329222 (+ 992us) server_negotiation.cc:407] Connection header received
0929 20:29:56.331133 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:56.331140 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:56.331275 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:56.331432 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:56.333726 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:56.334792 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:56.336716 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:56.337429 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:56.337579 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:56.341588 (+ 4009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:56.341596 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:56.341603 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:56.341657 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:56.344890 (+ 3233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:56.346067 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:56.346072 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:56.346074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:56.346232 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:56.346794 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:56.346804 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:56.346806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:56.347113 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:56.347278 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:56.348220 (+ 942us) server_negotiation.cc:299] Negotiation successful
0929 20:29:56.348461 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":275,"threads_started":1}
I20260929 20:29:56.349874 5669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:56.328243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.129:46533)
0929 20:29:56.329040 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:56.329081 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:56.330792 (+ 1711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:56.331785 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:56.331797 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:56.332592 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:56.333528 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:56.333558 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:56.334998 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:56.335003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:56.336553 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:56.336567 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:56.337678 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:56.337684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:56.337947 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:56.338700 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:56.338727 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:56.341286 (+ 2559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:56.345071 (+ 3785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:56.345107 (+ 36us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:56.345131 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:56.345890 (+ 759us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:56.346398 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:56.346414 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:56.346419 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:56.346655 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:56.347253 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:56.347262 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:56.347721 (+ 459us) client_negotiation.cc:769] Sending connection context
0929 20:29:56.348115 (+ 394us) client_negotiation.cc:240] Negotiation successful
0929 20:29:56.348696 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":301,"threads_started":1}
I20260929 20:29:56.351526 5666 heartbeater.cc:347] Connected to a master server at 127.28.244.190:45043
I20260929 20:29:56.351912 5666 heartbeater.cc:464] Registering TS with master...
I20260929 20:29:56.353003 5666 heartbeater.cc:511] Master 127.28.244.190:45043 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.244.130@KRBTEST.COM; defaulting to no policy
I20260929 20:29:56.355491 5477 ts_manager.cc:194] Registered new tserver with Master: 016699d1812448c4bb4fa4a17b00983f (127.28.244.129:44727)
I20260929 20:29:56.357106 5477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.129@KRBTEST.COM'} at 127.28.244.129:46533
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab.
Entry for principal kudu/127.28.244.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab.
I20260929 20:29:56.388305 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.130:0
--local_ip_for_outbound_sockets=127.28.244.130
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:45043
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:33671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:29:56.736063 5672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:56.736362 5672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:56.736469 5672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:56.744158 5672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:56.744278 5672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:56.744398 5672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.130
I20260929 20:29:56.754613 5672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:33671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:45043
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.130
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:56.755937 5672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:56.757274 5672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:56.771344 5677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:56.771451 5678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:56.771373 5680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:56.772498 5672 server_base.cc:1034] running on GCE node
I20260929 20:29:56.773154 5672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:56.774202 5672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:56.775483 5672 hybrid_clock.cc:648] HybridClock initialized: now 1790713796775433 us; error 43 us; skew 500 ppm
I20260929 20:29:56.775890 5672 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:56 dist-test-slave-p29l krb5kdc[5432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713796, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:56.783465 5672 init.cc:377] Logged in from keytab as kudu/127.28.244.130@KRBTEST.COM (short username kudu)
I20260929 20:29:56.785319 5672 webserver.cc:460] Webserver started at http://127.28.244.130:41011/ using document root <none> and password file <none>
I20260929 20:29:56.786119 5672 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:56.786273 5672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:56.786607 5672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:56.789963 5672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-1/data/instance:
uuid: "7eb0778f8afe4c80b4d98d442a6b90ce"
format_stamp: "Formatted at 2026-09-29 20:29:56 on dist-test-slave-p29l"
server_key: "78e77f3d49b5be340f9151800b7373f1"
server_key_iv: "0cc604db8037f3ba1ece8b8c62122a9c"
server_key_version: "encryptionkey@0"
I20260929 20:29:56.790786 5672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance:
uuid: "7eb0778f8afe4c80b4d98d442a6b90ce"
format_stamp: "Formatted at 2026-09-29 20:29:56 on dist-test-slave-p29l"
server_key: "78e77f3d49b5be340f9151800b7373f1"
server_key_iv: "0cc604db8037f3ba1ece8b8c62122a9c"
server_key_version: "encryptionkey@0"
I20260929 20:29:56.796298 5672 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 20:29:56.799872 5687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:56.801393 5672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:29:56.801534 5672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-1/wal
uuid: "7eb0778f8afe4c80b4d98d442a6b90ce"
format_stamp: "Formatted at 2026-09-29 20:29:56 on dist-test-slave-p29l"
server_key: "78e77f3d49b5be340f9151800b7373f1"
server_key_iv: "0cc604db8037f3ba1ece8b8c62122a9c"
server_key_version: "encryptionkey@0"
I20260929 20:29:56.801707 5672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:56.819237 5672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:56.823800 5672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:56.824132 5672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:56.825088 5672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:29:56.826421 5672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:29:56.826494 5672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:56.826649 5672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:29:56.826709 5672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:56.860579 5672 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.130:43797
I20260929 20:29:56.860669 5799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.130:43797 every 8 connection(s)
I20260929 20:29:56.861889 5672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
I20260929 20:29:56.871876 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 5672
I20260929 20:29:56.872074 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance
I20260929 20:29:56.872535 29650 external_mini_cluster.cc:1428] Setting key 52cd5517639f941e25bb7baa215959db
Sep 29 20:29:56 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713796, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:29:56.882467 5802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:56.864691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:36901 (local address 127.28.244.190:45043)
0929 20:29:56.865165 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:56.865170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:56.865910 (+ 740us) server_negotiation.cc:407] Connection header received
0929 20:29:56.867921 (+ 2011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:56.867926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:56.868028 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:56.868168 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:56.869660 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:56.870392 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:56.872017 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:56.872527 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:56.872659 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:56.876585 (+ 3926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:56.876591 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:56.876594 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:56.876626 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:56.879298 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:56.880110 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:56.880118 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:56.880122 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:56.880257 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:56.880693 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:56.880698 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:56.880701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:56.880980 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:56.881136 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:56.881832 (+ 696us) server_negotiation.cc:299] Negotiation successful
0929 20:29:56.882064 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":259,"threads_started":1}
I20260929 20:29:56.883457 5803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:56.865043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.130:36901)
0929 20:29:56.865733 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:56.865777 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:56.867612 (+ 1835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:56.868398 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:56.868407 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:56.868919 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:56.869485 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:56.869503 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:56.870558 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:56.870562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:56.871863 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:56.871883 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:56.872762 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:56.872770 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:56.873007 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:56.873785 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:56.873814 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:56.876344 (+ 2530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:56.879439 (+ 3095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:56.879451 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:56.879470 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:56.879991 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:56.880369 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:56.880374 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:56.880378 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:56.880575 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:56.881108 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:56.881116 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:56.881478 (+ 362us) client_negotiation.cc:769] Sending connection context
0929 20:29:56.881745 (+ 267us) client_negotiation.cc:240] Negotiation successful
0929 20:29:56.882305 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":265,"threads_started":1}
I20260929 20:29:56.885075 5800 heartbeater.cc:347] Connected to a master server at 127.28.244.190:45043
I20260929 20:29:56.885382 5800 heartbeater.cc:464] Registering TS with master...
I20260929 20:29:56.886121 5800 heartbeater.cc:511] Master 127.28.244.190:45043 requested a full tablet report, sending...
I20260929 20:29:56.887719 5477 ts_manager.cc:194] Registered new tserver with Master: 7eb0778f8afe4c80b4d98d442a6b90ce (127.28.244.130:43797)
I20260929 20:29:56.889032 5477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.130@KRBTEST.COM'} at 127.28.244.130:36901
WARNING: no policy specified for kudu/127.28.244.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab.
Entry for principal kudu/127.28.244.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab.
I20260929 20:29:56.923556 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.131:0
--local_ip_for_outbound_sockets=127.28.244.131
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:45043
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:33671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:29:57.263839 5806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:57.264186 5806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:57.264360 5806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:57.271726 5806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:57.271908 5806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:57.272161 5806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.131
I20260929 20:29:57.282162 5806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:33671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:45043
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.131
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:57.283501 5806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:57.284879 5806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:57.300348 5811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:57.300377 5812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:57.301716 5806 server_base.cc:1034] running on GCE node
W20260929 20:29:57.301874 5814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:57.302536 5806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:57.303676 5806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:57.304904 5806 hybrid_clock.cc:648] HybridClock initialized: now 1790713797304846 us; error 69 us; skew 500 ppm
I20260929 20:29:57.305279 5806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:57 dist-test-slave-p29l krb5kdc[5432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713797, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:57.308836 5806 init.cc:377] Logged in from keytab as kudu/127.28.244.131@KRBTEST.COM (short username kudu)
I20260929 20:29:57.310511 5806 webserver.cc:460] Webserver started at http://127.28.244.131:36889/ using document root <none> and password file <none>
I20260929 20:29:57.311215 5806 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:57.311374 5806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:57.311699 5806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:29:57.314854 5806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-2/data/instance:
uuid: "6b3a7c011c874bfdb971775b226c57ea"
format_stamp: "Formatted at 2026-09-29 20:29:57 on dist-test-slave-p29l"
server_key: "cfee5b02daeae17eb3b93b4cf1b31cfd"
server_key_iv: "4aa041b191440a5e6c660d69ed21dadf"
server_key_version: "encryptionkey@0"
I20260929 20:29:57.315671 5806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance:
uuid: "6b3a7c011c874bfdb971775b226c57ea"
format_stamp: "Formatted at 2026-09-29 20:29:57 on dist-test-slave-p29l"
server_key: "cfee5b02daeae17eb3b93b4cf1b31cfd"
server_key_iv: "4aa041b191440a5e6c660d69ed21dadf"
server_key_version: "encryptionkey@0"
I20260929 20:29:57.321194 5806 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260929 20:29:57.325012 5821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:57.326459 5806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:29:57.326642 5806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-2/wal
uuid: "6b3a7c011c874bfdb971775b226c57ea"
format_stamp: "Formatted at 2026-09-29 20:29:57 on dist-test-slave-p29l"
server_key: "cfee5b02daeae17eb3b93b4cf1b31cfd"
server_key_iv: "4aa041b191440a5e6c660d69ed21dadf"
server_key_version: "encryptionkey@0"
I20260929 20:29:57.326848 5806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:57.360340 5666 heartbeater.cc:503] Master 127.28.244.190:45043 was elected leader, sending a full tablet report...
I20260929 20:29:57.365655 5806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:57.370256 5806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:57.370586 5806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:57.371657 5806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:29:57.373169 5806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:29:57.373281 5806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:57.373433 5806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:29:57.373510 5806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:57.411047 5806 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.131:36833
I20260929 20:29:57.411168 5933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.131:36833 every 8 connection(s)
I20260929 20:29:57.412520 5806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
I20260929 20:29:57.416309 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 5806
I20260929 20:29:57.416501 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance
I20260929 20:29:57.417088 29650 external_mini_cluster.cc:1428] Setting key e5c47128f0c0cb5499931166db9936d7
Sep 29 20:29:57 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713797, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:29:57.434214 5936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:57.415342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:60005 (local address 127.28.244.190:45043)
0929 20:29:57.415858 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:57.415865 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:57.416961 (+ 1096us) server_negotiation.cc:407] Connection header received
0929 20:29:57.419433 (+ 2472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:57.419442 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:57.419560 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:57.419675 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:57.421476 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:57.422292 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:57.423796 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:57.424258 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:57.424355 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:57.427898 (+ 3543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:57.427903 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:57.427907 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:57.427950 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:57.430824 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:57.431774 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:57.431780 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:57.431783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:57.431958 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:57.432525 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:57.432530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:57.432533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:57.432911 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:57.433066 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:57.433780 (+ 714us) server_negotiation.cc:299] Negotiation successful
0929 20:29:57.433951 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":257,"threads_started":1}
I20260929 20:29:57.435240 5937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:57.415928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.131:60005)
0929 20:29:57.416739 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:57.416796 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:57.419067 (+ 2271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:57.419960 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:57.419973 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:57.420683 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:57.421267 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:57.421284 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:57.422436 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:57.422440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:57.423659 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:57.423670 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:57.424539 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:57.424545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:57.424754 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:57.425428 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:57.425470 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:57.427638 (+ 2168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:29:57.430972 (+ 3334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:57.430983 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:57.431002 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:57.431658 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:57.432076 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:29:57.432082 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:29:57.432086 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:29:57.432354 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:29:57.433039 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:29:57.433046 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:29:57.433445 (+ 399us) client_negotiation.cc:769] Sending connection context
0929 20:29:57.433687 (+ 242us) client_negotiation.cc:240] Negotiation successful
0929 20:29:57.434075 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":293,"threads_started":1}
I20260929 20:29:57.436723 5934 heartbeater.cc:347] Connected to a master server at 127.28.244.190:45043
I20260929 20:29:57.437021 5934 heartbeater.cc:464] Registering TS with master...
I20260929 20:29:57.437719 5934 heartbeater.cc:511] Master 127.28.244.190:45043 requested a full tablet report, sending...
I20260929 20:29:57.439124 5477 ts_manager.cc:194] Registered new tserver with Master: 6b3a7c011c874bfdb971775b226c57ea (127.28.244.131:36833)
I20260929 20:29:57.439980 5477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.131@KRBTEST.COM'} at 127.28.244.131:60005
I20260929 20:29:57.445500 29650 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.244.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.244.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.244.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.129.keytab.
Entry for principal oryx/127.28.244.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.129.keytab.
WARNING: no policy specified for oryx/127.28.244.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.244.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.244.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.130.keytab.
Entry for principal oryx/127.28.244.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.130.keytab.
WARNING: no policy specified for oryx/127.28.244.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.244.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.244.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.131.keytab.
Entry for principal oryx/127.28.244.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.131.keytab.
I20260929 20:29:57.582437 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 5538
Sep 29 20:29:57 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.129@KRBTEST.COM
I20260929 20:29:57.598737 5945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:57.583496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34598 (local address 127.28.244.129:44727)
0929 20:29:57.584114 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:57.584122 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:57.584150 (+ 28us) server_negotiation.cc:407] Connection header received
0929 20:29:57.584309 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:57.584316 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:57.584567 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:57.584707 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:57.586030 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:57.587061 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:57.588337 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:57.588904 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:57.589006 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:57.593180 (+ 4174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:57.593194 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:57.593208 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:57.593254 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:57.596217 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:57.596843 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:57.596849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:57.596852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:57.596983 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:57.597334 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:57.597337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:57.597339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:57.597683 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:57.597902 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:57.598231 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 20:29:57.598426 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":323,"threads_started":1}
I20260929 20:29:57.600056 5661 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:29:57.684520 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 5538
I20260929 20:29:57.701562 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 5672
Sep 29 20:29:57 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.130@KRBTEST.COM
I20260929 20:29:57.718315 5947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:57.702469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38842 (local address 127.28.244.130:43797)
0929 20:29:57.703082 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:57.703091 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:57.703121 (+ 30us) server_negotiation.cc:407] Connection header received
0929 20:29:57.703188 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:57.703197 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:57.703477 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:57.703647 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:57.704953 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:57.705998 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:57.707230 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:57.707897 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:57.707995 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:57.711974 (+ 3979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:57.711985 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:57.712000 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:57.712040 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:57.715324 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:57.716315 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:57.716322 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:57.716324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:57.716493 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:57.716849 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:57.716852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:57.716855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:57.717190 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:57.717447 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:57.717694 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 20:29:57.717974 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":288,"threads_started":1}
I20260929 20:29:57.719619 5795 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:29:57.801824 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 5672
I20260929 20:29:57.818718 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 5806
Sep 29 20:29:57 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.131@KRBTEST.COM
I20260929 20:29:57.835436 5949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:57.819740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38762 (local address 127.28.244.131:36833)
0929 20:29:57.820391 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:57.820398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:57.820427 (+ 29us) server_negotiation.cc:407] Connection header received
0929 20:29:57.820496 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:57.820503 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:57.820716 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:57.820843 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:57.822136 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:57.823195 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:57.824661 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:57.825223 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:57.825324 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:57.829809 (+ 4485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:57.829820 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:57.829841 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:57.829881 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:57.832849 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:57.833559 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:57.833565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:57.833567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:57.833696 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:57.834067 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:57.834070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:57.834072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:57.834422 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:57.834657 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:57.834940 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 20:29:57.835147 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":345,"threads_started":1}
I20260929 20:29:57.836901 5929 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:29:57.919358 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 5806
I20260929 20:29:57.935950 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 5445
I20260929 20:29:57.937141 5507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:29:58.042203 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 5445
I20260929 20:29:58.063697 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:45043
--webserver_interface=127.28.244.190
--webserver_port=33821
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:33671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:45043
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:29:58.394227 5952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:58.394501 5952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:58.394665 5952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:58.401844 5952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:29:58.401952 5952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:58.402212 5952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:58.402348 5952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:29:58.402428 5952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:29:58.412608 5952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:33671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:45043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:45043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=33821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:58.414003 5952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:58.415386 5952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:58.427858 5960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:58.427943 5958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:58.427896 5957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:58.433596 5952 server_base.cc:1034] running on GCE node
I20260929 20:29:58.434453 5952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:58.436046 5952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:58.437291 5952 hybrid_clock.cc:648] HybridClock initialized: now 1790713798437238 us; error 43 us; skew 500 ppm
I20260929 20:29:58.437716 5952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:58 dist-test-slave-p29l krb5kdc[5432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713798, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:58.440987 5952 init.cc:377] Logged in from keytab as kudu/127.28.244.190@KRBTEST.COM (short username kudu)
I20260929 20:29:58.442579 5952 webserver.cc:460] Webserver started at http://127.28.244.190:33821/ using document root <none> and password file <none>
I20260929 20:29:58.443279 5952 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:58.443394 5952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:58.448916 5952 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260929 20:29:58.451561 5967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:58.452970 5952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:29:58.453100 5952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "ca7f9fb997004ee4a4710604dcfc1e93"
format_stamp: "Formatted at 2026-09-29 20:29:55 on dist-test-slave-p29l"
server_key: "0f3d05225474e55eba507152e2ec5f0e"
server_key_iv: "6f15e62ff112aec2a62e0d2a671486ee"
server_key_version: "encryptionkey@0"
I20260929 20:29:58.454005 5952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:58.471220 5952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:58.475736 5952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:58.475997 5952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:58.494596 5952 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:45043
I20260929 20:29:58.494614 6018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:45043 every 8 connection(s)
I20260929 20:29:58.496183 5952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:29:58.503139 6019 sys_catalog.cc:263] Verifying existing consensus state
I20260929 20:29:58.505738 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 5952
I20260929 20:29:58.506752 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.129:44727
--local_ip_for_outbound_sockets=127.28.244.129
--tserver_master_addrs=127.28.244.190:45043
--webserver_port=34213
--webserver_interface=127.28.244.129
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:33671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.244.129
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:29:58.506942 6019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93: Bootstrap starting.
I20260929 20:29:58.528599 6019 log.cc:824] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93: Log is configured to *not* fsync() on all Append() calls
I20260929 20:29:58.537827 6019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:29:58.538385 6019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93: Bootstrap complete.
I20260929 20:29:58.547570 6019 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7f9fb997004ee4a4710604dcfc1e93" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 45043 } }
I20260929 20:29:58.548520 6019 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca7f9fb997004ee4a4710604dcfc1e93, State: Initialized, Role: FOLLOWER
I20260929 20:29:58.549156 6019 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7f9fb997004ee4a4710604dcfc1e93" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 45043 } }
I20260929 20:29:58.549400 6019 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:29:58.549536 6019 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:29:58.549722 6019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93 [term 1 FOLLOWER]: Advancing to term 2
I20260929 20:29:58.553720 6019 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7f9fb997004ee4a4710604dcfc1e93" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 45043 } }
I20260929 20:29:58.554301 6019 leader_election.cc:302] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca7f9fb997004ee4a4710604dcfc1e93; no voters:
I20260929 20:29:58.554991 6019 leader_election.cc:288] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 20:29:58.555125 6023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 20:29:58.556586 6023 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93 [term 2 LEADER]: Becoming Leader. State: Replica: ca7f9fb997004ee4a4710604dcfc1e93, State: Running, Role: LEADER
I20260929 20:29:58.557274 6023 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7f9fb997004ee4a4710604dcfc1e93" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 45043 } }
I20260929 20:29:58.557523 6019 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:29:58.560042 6025 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca7f9fb997004ee4a4710604dcfc1e93. Latest consensus state: current_term: 2 leader_uuid: "ca7f9fb997004ee4a4710604dcfc1e93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7f9fb997004ee4a4710604dcfc1e93" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 45043 } } }
I20260929 20:29:58.560415 6025 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93 [sys.catalog]: This master's current role is: LEADER
I20260929 20:29:58.560940 6024 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ca7f9fb997004ee4a4710604dcfc1e93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7f9fb997004ee4a4710604dcfc1e93" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 45043 } } }
I20260929 20:29:58.561244 6024 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93 [sys.catalog]: This master's current role is: LEADER
I20260929 20:29:58.569487 6029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:29:58.576284 6029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:29:58.577740 6029 catalog_manager.cc:1259] Loaded cluster ID: 0057c5066ef5422baab535de48ec7719
I20260929 20:29:58.577924 6029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:29:58.581925 6029 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:29:58.583401 6029 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ca7f9fb997004ee4a4710604dcfc1e93: Loaded TSK: 0
I20260929 20:29:58.585083 6029 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 20:29:58.853525 6021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:58.853809 6021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:58.853963 6021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:58.861726 6021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:58.861835 6021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:58.862020 6021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.129
I20260929 20:29:58.872381 6021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:33671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.129.keytab
--principal=oryx/127.28.244.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.129:44727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.244.129
--webserver_port=34213
--tserver_master_addrs=127.28.244.190:45043
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.129
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:58.873754 6021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:58.875052 6021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:58.889210 6045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:58.889273 6046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:58.889243 6048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:58.891244 6021 server_base.cc:1034] running on GCE node
I20260929 20:29:58.892027 6021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:58.893239 6021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:58.894490 6021 hybrid_clock.cc:648] HybridClock initialized: now 1790713798894430 us; error 50 us; skew 500 ppm
I20260929 20:29:58.894876 6021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:58 dist-test-slave-p29l krb5kdc[5432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713798, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.244.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:58.898864 6021 init.cc:377] Logged in from keytab as oryx/127.28.244.129@KRBTEST.COM (short username oryx)
I20260929 20:29:58.900676 6021 webserver.cc:460] Webserver started at http://127.28.244.129:34213/ using document root <none> and password file <none>
I20260929 20:29:58.901610 6021 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:58.901753 6021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:58.908962 6021 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:29:58.912299 6055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:58.913939 6021 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 20:29:58.914130 6021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-0/wal
uuid: "016699d1812448c4bb4fa4a17b00983f"
format_stamp: "Formatted at 2026-09-29 20:29:56 on dist-test-slave-p29l"
server_key: "2f66a0cfad152050cbda4592f01f9ba1"
server_key_iv: "a314c84e469aa7b135ad9db50e886363"
server_key_version: "encryptionkey@0"
I20260929 20:29:58.915426 6021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:58.940927 6021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:58.944875 6021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:58.945153 6021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:58.946103 6021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:29:58.947422 6021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:29:58.947553 6021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:58.947677 6021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:29:58.947763 6021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:58.981141 6021 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.129:44727
I20260929 20:29:58.981225 6167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.129:44727 every 8 connection(s)
I20260929 20:29:58.982596 6021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
I20260929 20:29:58.992398 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 6021
I20260929 20:29:58.993170 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.130:43797
--local_ip_for_outbound_sockets=127.28.244.130
--tserver_master_addrs=127.28.244.190:45043
--webserver_port=41011
--webserver_interface=127.28.244.130
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:33671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.244.130
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:29:58 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.129@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:29:58 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.129@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:29:59.001973 6171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:58.985605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.129:52419)
0929 20:29:58.986421 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:58.986474 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:58.988247 (+ 1773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:58.989552 (+ 1305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:58.989559 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:58.990063 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:58.990600 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:58.990616 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:58.992252 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:58.992259 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:58.994405 (+ 2146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:58.994425 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:58.995448 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:58.995455 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:58.995754 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:58.996533 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:58.996568 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:59.000982 (+ 4414us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":351,"threads_started":1}
W20260929 20:29:59.002635 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:29:58.985615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:52419 (local address 127.28.244.190:45043)
0929 20:29:58.986271 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:58.986283 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:58.986601 (+ 318us) server_negotiation.cc:407] Connection header received
0929 20:29:58.988743 (+ 2142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:58.988756 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:58.989090 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:58.989381 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:58.990780 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:58.992087 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:58.994573 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:58.995220 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:58.995347 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:59.001457 (+ 6110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.129:52419: BlockingRecv error: recv got EOF from 127.28.244.129:52419 (error 108)
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":261,"threads_started":1}
W20260929 20:29:59.002662 6168 heartbeater.cc:644] Failed to heartbeat to 127.28.244.190:45043 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.244.190:45043: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Sep 29 20:29:59 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.129@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:29:59 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.129@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 20:29:59.013072 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:29:59.003693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:59611 (local address 127.28.244.190:45043)
0929 20:29:59.003882 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:59.003887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:59.003910 (+ 23us) server_negotiation.cc:407] Connection header received
0929 20:29:59.003960 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:59.003964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:59.004028 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:59.004108 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:59.004996 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:59.005746 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:59.007739 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:59.008209 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:59.008303 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:59.012903 (+ 4600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.129:59611: BlockingRecv error: recv got EOF from 127.28.244.129:59611 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 20:29:59.013118 6171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:59.003251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.129:59611)
0929 20:29:59.003475 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:59.003509 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:59.003634 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:59.004172 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:59.004179 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:59.004611 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:59.004841 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:59.004855 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:59.005897 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:59.005906 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:59.007591 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:59.007608 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:59.008358 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:59.008366 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:59.008581 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:59.009189 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:59.009216 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:59.012727 (+ 3511us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":68}
Sep 29 20:29:59 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.129@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:29:59 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.129@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:29:59.021673 6171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:59.013924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.129:54361)
0929 20:29:59.014111 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:59.014130 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:59.014241 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:59.014582 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:59.014586 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:59.014892 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:59.015035 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:59.015043 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:59.016097 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:59.016109 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:59.017004 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:59.017015 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:59.017820 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:59.017826 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:59.017995 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:59.018613 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:59.018629 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:59.021356 (+ 2727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":59}
W20260929 20:29:59.021883 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:29:59.014011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:54361 (local address 127.28.244.190:45043)
0929 20:29:59.014168 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:59.014174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:59.014191 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:29:59.014369 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:59.014373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:59.014450 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:59.014559 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:59.015170 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:59.015950 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:59.017149 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:59.017669 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:59.017799 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:59.021578 (+ 3779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.129:54361: BlockingRecv error: recv got EOF from 127.28.244.129:54361 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 20:29:59.022034 6168 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 20:29:59.326385 6172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:59.326691 6172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:59.326784 6172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:59.334112 6172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:59.334239 6172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:59.334352 6172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.130
I20260929 20:29:59.345216 6172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:33671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.130.keytab
--principal=oryx/127.28.244.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.130:43797
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.244.130
--webserver_port=41011
--tserver_master_addrs=127.28.244.190:45043
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.130
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:59.346598 6172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:59.347924 6172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:59.361649 6177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:59.361716 6180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:59.361816 6178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:59.362620 6172 server_base.cc:1034] running on GCE node
I20260929 20:29:59.363471 6172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:59.364564 6172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:59.365785 6172 hybrid_clock.cc:648] HybridClock initialized: now 1790713799365733 us; error 49 us; skew 500 ppm
I20260929 20:29:59.366146 6172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:59 dist-test-slave-p29l krb5kdc[5432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713799, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.244.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:59.369524 6172 init.cc:377] Logged in from keytab as oryx/127.28.244.130@KRBTEST.COM (short username oryx)
I20260929 20:29:59.371160 6172 webserver.cc:460] Webserver started at http://127.28.244.130:41011/ using document root <none> and password file <none>
I20260929 20:29:59.371903 6172 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:59.372018 6172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:59.377588 6172 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 20:29:59.380424 6187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:59.381809 6172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:29:59.381999 6172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-1/wal
uuid: "7eb0778f8afe4c80b4d98d442a6b90ce"
format_stamp: "Formatted at 2026-09-29 20:29:56 on dist-test-slave-p29l"
server_key: "78e77f3d49b5be340f9151800b7373f1"
server_key_iv: "0cc604db8037f3ba1ece8b8c62122a9c"
server_key_version: "encryptionkey@0"
I20260929 20:29:59.383101 6172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:59.401531 6172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:59.405936 6172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:59.406225 6172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:59.407236 6172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:29:59.408697 6172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:29:59.408820 6172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:59.408946 6172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:29:59.409013 6172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:59.442903 6172 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.130:43797
I20260929 20:29:59.442960 6299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.130:43797 every 8 connection(s)
I20260929 20:29:59.444471 6172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
I20260929 20:29:59.450007 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 6172
I20260929 20:29:59.450719 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.131:36833
--local_ip_for_outbound_sockets=127.28.244.131
--tserver_master_addrs=127.28.244.190:45043
--webserver_port=36889
--webserver_interface=127.28.244.131
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:33671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.244.131
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:29:59 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.130@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:29:59 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.130@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 20:29:59.462922 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:29:59.447172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:41811 (local address 127.28.244.190:45043)
0929 20:29:59.447404 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:59.447410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:59.448643 (+ 1233us) server_negotiation.cc:407] Connection header received
0929 20:29:59.450915 (+ 2272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:59.450923 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:59.451012 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:59.451120 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:59.453301 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:59.454023 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:59.456528 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:59.456954 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:59.457049 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:59.462617 (+ 5568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.130:41811: BlockingRecv error: recv got EOF from 127.28.244.130:41811 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 20:29:59.463399 6302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:59.447689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.130:41811)
0929 20:29:59.448485 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:59.448523 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:59.450565 (+ 2042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:59.451422 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:59.451437 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:59.452274 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:59.453115 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:59.453143 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:59.454195 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:59.454204 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:59.456381 (+ 2177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:59.456398 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:59.457171 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:59.457180 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:59.457566 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:59.458398 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:59.458454 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:59.462335 (+ 3881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":312,"threads_started":1}
W20260929 20:29:59.464078 6300 heartbeater.cc:644] Failed to heartbeat to 127.28.244.190:45043 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.244.190:45043: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Sep 29 20:29:59 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.130@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:29:59 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.130@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:29:59.473994 6302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:59.464658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.130:46899)
0929 20:29:59.464894 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:59.464916 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:59.465025 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:59.465308 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:59.465315 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:59.465760 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:59.465978 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:59.465992 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:59.467023 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:59.467032 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:59.468775 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:59.468793 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:59.469507 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:59.469515 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:59.469779 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:59.470432 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:59.470472 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:59.473781 (+ 3309us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":82}
W20260929 20:29:59.474408 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:29:59.464874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:46899 (local address 127.28.244.190:45043)
0929 20:29:59.465033 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:59.465037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:59.465053 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:29:59.465122 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:59.465125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:59.465181 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:59.465252 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:59.466137 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:59.466863 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:59.468934 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:59.469336 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:59.469448 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:59.474100 (+ 4652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.130:46899: BlockingRecv error: recv got EOF from 127.28.244.130:46899 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 20:29:59 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.130@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:29:59 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.130@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:29:59.483909 6302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:59.474770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.130:35531)
0929 20:29:59.474971 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:59.474996 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:59.475108 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:59.475435 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:59.475443 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:59.475923 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:59.476157 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:59.476173 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:59.477142 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:59.477148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:59.478626 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:59.478640 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:59.479298 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:59.479304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:59.479521 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:59.480229 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:59.480252 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:59.483576 (+ 3324us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260929 20:29:59.484081 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:29:59.474857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:35531 (local address 127.28.244.190:45043)
0929 20:29:59.475013 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:59.475017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:59.475036 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:29:59.475232 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:59.475236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:59.475290 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:59.475367 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:59.476299 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:59.477009 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:59.478768 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:59.479160 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:59.479255 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:59.483757 (+ 4502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.130:35531: BlockingRecv error: recv got EOF from 127.28.244.130:35531 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 20:29:59.484535 6300 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 20:29:59.790961 6303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:29:59.791297 6303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:29:59.791474 6303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:29:59.798846 6303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:29:59.799003 6303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:29:59.799186 6303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.131
I20260929 20:29:59.809175 6303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:33671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/krb5kdc/oryx_127.28.244.131.keytab
--principal=oryx/127.28.244.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.131:36833
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.244.131
--webserver_port=36889
--tserver_master_addrs=127.28.244.190:45043
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.131
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:29:59.810518 6303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:29:59.811848 6303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:29:59.825254 6311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:59.825285 6308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:29:59.825503 6309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:29:59.826129 6303 server_base.cc:1034] running on GCE node
I20260929 20:29:59.826833 6303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:29:59.828145 6303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:29:59.829432 6303 hybrid_clock.cc:648] HybridClock initialized: now 1790713799829353 us; error 69 us; skew 500 ppm
I20260929 20:29:59.829824 6303 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:29:59 dist-test-slave-p29l krb5kdc[5432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713799, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.244.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:29:59.833992 6303 init.cc:377] Logged in from keytab as oryx/127.28.244.131@KRBTEST.COM (short username oryx)
I20260929 20:29:59.836125 6303 webserver.cc:460] Webserver started at http://127.28.244.131:36889/ using document root <none> and password file <none>
I20260929 20:29:59.836921 6303 fs_manager.cc:359] Metadata directory not provided
I20260929 20:29:59.837049 6303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:29:59.842604 6303 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 20:29:59.845419 6318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:59.846848 6303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:29:59.847069 6303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-2/wal
uuid: "6b3a7c011c874bfdb971775b226c57ea"
format_stamp: "Formatted at 2026-09-29 20:29:57 on dist-test-slave-p29l"
server_key: "cfee5b02daeae17eb3b93b4cf1b31cfd"
server_key_iv: "4aa041b191440a5e6c660d69ed21dadf"
server_key_version: "encryptionkey@0"
I20260929 20:29:59.848104 6303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:29:59.861537 6303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:29:59.866016 6303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:29:59.866294 6303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:29:59.867244 6303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:29:59.868630 6303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:29:59.868757 6303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:59.868882 6303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:29:59.868976 6303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:29:59.902004 6303 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.131:36833
I20260929 20:29:59.902093 6430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.131:36833 every 8 connection(s)
I20260929 20:29:59.903491 6303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
I20260929 20:29:59.906828 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 6303
Sep 29 20:29:59 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.131@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:29:59 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.131@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 20:29:59.920251 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:29:59.906212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:42899 (local address 127.28.244.190:45043)
0929 20:29:59.906486 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:59.906493 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:59.907729 (+ 1236us) server_negotiation.cc:407] Connection header received
0929 20:29:59.910019 (+ 2290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:59.910028 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:59.910123 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:59.910234 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:59.911712 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:59.912519 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:59.914012 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:59.914412 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:59.914541 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:59.919950 (+ 5409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.131:42899: BlockingRecv error: recv got EOF from 127.28.244.131:42899 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:29:59.920917 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:59.906708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.131:42899)
0929 20:29:59.907541 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:59.907581 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:59.909755 (+ 2174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:59.910476 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:59.910484 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:59.910973 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:59.911534 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:59.911550 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:59.912664 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:59.912676 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:59.913861 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:59.913874 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:59.914626 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:59.914633 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:59.914854 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:59.915502 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:59.915523 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:59.919577 (+ 4054us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":327,"threads_started":1}
W20260929 20:29:59.921712 6431 heartbeater.cc:644] Failed to heartbeat to 127.28.244.190:45043 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.244.190:45043: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
I20260929 20:29:59.924777 6435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:59.907941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33412 (local address 127.28.244.190:45043)
0929 20:29:59.908891 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:59.908896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:59.908909 (+ 13us) server_negotiation.cc:407] Connection header received
0929 20:29:59.908958 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:59.908962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:59.909023 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:59.909146 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:59.910918 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:59.911696 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:59.913015 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:59.913410 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:59.913547 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:59.916556 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:29:59.916568 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:29:59.916589 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:29:59.916658 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:29:59.920753 (+ 4095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:59.921615 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:59.921624 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:59.921635 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:59.921933 (+ 298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:29:59.922445 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:29:59.922451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:29:59.922454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:29:59.922982 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:29:59.923336 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0929 20:29:59.923962 (+ 626us) server_negotiation.cc:299] Negotiation successful
0929 20:29:59.924338 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":868,"thread_start_us":782,"threads_started":1}
Sep 29 20:29:59 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.131@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:29:59 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.131@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:29:59.931650 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:59.923080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.131:36439)
0929 20:29:59.923436 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:59.923452 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:59.923576 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:59.924008 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:59.924012 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:59.924413 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:59.924591 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:59.924602 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:59.925854 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:59.925860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:59.926916 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:59.926934 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:59.927775 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:59.927783 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:59.928034 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:59.928679 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:59.928696 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:59.931438 (+ 2742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":69}
W20260929 20:29:59.932042 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:29:59.923204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:36439 (local address 127.28.244.190:45043)
0929 20:29:59.923358 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:59.923365 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:59.923518 (+ 153us) server_negotiation.cc:407] Connection header received
0929 20:29:59.923743 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:59.923749 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:59.923859 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:59.923976 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:59.924740 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:59.925703 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:59.927082 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:59.927599 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:59.927745 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:59.931698 (+ 3953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.131:36439: BlockingRecv error: recv got EOF from 127.28.244.131:36439 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 29 20:29:59 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.131@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:29:59 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.131@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:29:59.940418 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:29:59.932452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.131:41193)
0929 20:29:59.932643 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 20:29:59.932670 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 20:29:59.932756 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:29:59.933077 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:29:59.933080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:29:59.933362 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:29:59.933519 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:59.933527 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:59.934543 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:59.934547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:59.935525 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:29:59.935538 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:29:59.936297 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:29:59.936301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:29:59.936488 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:59.937038 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:29:59.937054 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:29:59.940052 (+ 2998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":69}
W20260929 20:29:59.940680 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:29:59.932613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:41193 (local address 127.28.244.190:45043)
0929 20:29:59.932825 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 20:29:59.932828 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 20:29:59.932844 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:29:59.932888 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:29:59.932891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:29:59.932943 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:29:59.933034 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:29:59.933650 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:59.934403 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:59.935661 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:29:59.936145 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:29:59.936243 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:29:59.940441 (+ 4198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.131:41193: BlockingRecv error: recv got EOF from 127.28.244.131:41193 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
W20260929 20:29:59.940811 6431 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 20:30:00 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.129@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:00 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.129@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:00.031261 6436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:00.022593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.129:38599)
0929 20:30:00.023142 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:00.023159 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:00.023277 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:00.023598 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:00.023601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:00.024120 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:00.024307 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:00.024315 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:00.025657 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:00.025661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:00.026616 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:00.026628 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:00.027388 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:00.027394 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:00.027589 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:00.028219 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:00.028236 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:00.031032 (+ 2796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":289,"threads_started":1}
W20260929 20:30:00.031548 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:00.022762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:38599 (local address 127.28.244.190:45043)
0929 20:30:00.022935 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:00.022939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:00.023237 (+ 298us) server_negotiation.cc:407] Connection header received
0929 20:30:00.023388 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:00.023391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:00.023462 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:00.023554 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:00.024471 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:00.025370 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:00.026768 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:00.027206 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:00.027341 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:00.031281 (+ 3940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.129:38599: BlockingRecv error: recv got EOF from 127.28.244.129:38599 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 20:30:00 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.130@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:00 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.130@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:00.493460 6437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:00.485228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.130:46489)
0929 20:30:00.485768 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:00.485785 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:00.485899 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:00.486277 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:00.486280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:00.486704 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:00.486944 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:00.486954 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:00.488089 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:00.488092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:00.489042 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:00.489054 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:00.489732 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:00.489736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:00.489856 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:00.490418 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:00.490435 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:00.493191 (+ 2756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":286,"threads_started":1}
W20260929 20:30:00.493958 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:00.485298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:46489 (local address 127.28.244.190:45043)
0929 20:30:00.485501 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:00.485509 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:00.485862 (+ 353us) server_negotiation.cc:407] Connection header received
0929 20:30:00.486003 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:00.486007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:00.486069 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:00.486204 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:00.487073 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:00.487957 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:00.489173 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:00.489572 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:00.489662 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:00.493672 (+ 4010us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.130:46489: BlockingRecv error: recv got EOF from 127.28.244.130:46489 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 20:30:00 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.131@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:00 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.131@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:00.950309 6438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:00.941542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.131:41815)
0929 20:30:00.942122 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:00.942149 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:00.942297 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:00.942697 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:00.942700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:00.943133 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:00.943327 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:00.943336 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:00.944458 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:00.944462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:00.945408 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:00.945420 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:00.946131 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:00.946136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:00.946267 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:00.946906 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:00.946924 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:00.949971 (+ 3047us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":283,"threads_started":1}
W20260929 20:30:00.950397 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:00.941640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:41815 (local address 127.28.244.190:45043)
0929 20:30:00.941903 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:00.941911 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:00.942244 (+ 333us) server_negotiation.cc:407] Connection header received
0929 20:30:00.942502 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:00.942506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:00.942568 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:00.942647 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:00.943531 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:00.944313 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:00.945543 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:00.945942 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:00.946040 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:00.950188 (+ 4148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.131:41815: BlockingRecv error: recv got EOF from 127.28.244.131:41815 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 29 20:30:01 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.129@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:01 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.129@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:01.040983 6439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:01.032358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.129:34155)
0929 20:30:01.032903 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:01.032921 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:01.033079 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:01.033451 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:01.033457 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:01.033868 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:01.034036 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:01.034046 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:01.035123 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:01.035127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:01.036485 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:01.036506 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:01.037321 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:01.037325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:01.037437 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:01.038033 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:01.038051 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:01.040748 (+ 2697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":310,"threads_started":1}
W20260929 20:30:01.041184 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:01.032451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:34155 (local address 127.28.244.190:45043)
0929 20:30:01.032738 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:01.032744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:01.033006 (+ 262us) server_negotiation.cc:407] Connection header received
0929 20:30:01.033218 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:01.033223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:01.033283 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:01.033385 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:01.034178 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:01.034968 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:01.036636 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:01.037178 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:01.037277 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:01.040947 (+ 3670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.129:34155: BlockingRecv error: recv got EOF from 127.28.244.129:34155 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 29 20:30:01 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.130@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:01 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.130@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:01.503985 6440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:01.494567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.130:38017)
0929 20:30:01.495197 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:01.495213 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:01.495324 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:01.495701 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:01.495705 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:01.496161 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:01.496333 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:01.496342 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:01.497756 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:01.497759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:01.498645 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:01.498655 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:01.499592 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:01.499596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:01.499704 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:01.500342 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:01.500359 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:01.503599 (+ 3240us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":377,"threads_started":1}
W20260929 20:30:01.504094 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:01.494674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:38017 (local address 127.28.244.190:45043)
0929 20:30:01.494862 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:01.494868 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:01.495296 (+ 428us) server_negotiation.cc:407] Connection header received
0929 20:30:01.495435 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:01.495440 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:01.495558 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:01.495665 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:01.496476 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:01.497615 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:01.498786 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:01.499443 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:01.499567 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:01.503799 (+ 4232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.130:38017: BlockingRecv error: recv got EOF from 127.28.244.130:38017 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 20:30:01 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.131@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:01 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.131@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:01.964314 6441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:01.951216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.131:51385)
0929 20:30:01.951797 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:01.951841 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:01.951954 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:01.952236 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:01.952239 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:01.952751 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:01.952974 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:01.952985 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:01.953981 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:01.953985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:01.954866 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:01.954876 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:01.955606 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:01.955610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:01.955735 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:01.961148 (+ 5413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:01.961167 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:01.964065 (+ 2898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":347,"threads_started":1}
W20260929 20:30:01.964570 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:01.951436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:51385 (local address 127.28.244.190:45043)
0929 20:30:01.951605 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:01.951609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:01.951914 (+ 305us) server_negotiation.cc:407] Connection header received
0929 20:30:01.952052 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:01.952055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:01.952115 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:01.952197 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:01.953111 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:01.953868 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:01.955020 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:01.955445 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:01.955548 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:01.964256 (+ 8708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.131:51385: BlockingRecv error: recv got EOF from 127.28.244.131:51385 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 20:30:02 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.129@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:02 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.129@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:02.051390 6442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:02.042018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.129:48559)
0929 20:30:02.042727 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:02.042751 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:02.042914 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:02.043440 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:02.043446 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:02.043914 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:02.044101 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:02.044109 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:02.045407 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:02.045412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:02.046730 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:02.046749 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:02.047567 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:02.047572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:02.047712 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:02.048350 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:02.048367 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:02.051140 (+ 2773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":422,"threads_started":1}
W20260929 20:30:02.051587 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:02.042186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:48559 (local address 127.28.244.190:45043)
0929 20:30:02.042567 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:02.042577 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:02.042860 (+ 283us) server_negotiation.cc:407] Connection header received
0929 20:30:02.043095 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:02.043099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:02.043163 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:02.043357 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:02.044259 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:02.045089 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:02.046933 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:02.047397 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:02.047501 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:02.051360 (+ 3859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.129:48559: BlockingRecv error: recv got EOF from 127.28.244.129:48559 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 29 20:30:02 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.130@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:02 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.130@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:02.513278 6443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:02.505039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.130:51751)
0929 20:30:02.505596 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:02.505612 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:02.505772 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:02.506132 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:02.506137 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:02.506493 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:02.506660 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:02.506668 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:02.507713 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:02.507720 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:02.508634 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:02.508645 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:02.509346 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:02.509354 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:02.509557 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:02.510153 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:02.510170 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:02.512882 (+ 2712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":280,"threads_started":1}
W20260929 20:30:02.513334 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:02.505198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:51751 (local address 127.28.244.190:45043)
0929 20:30:02.505385 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:02.505391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:02.505705 (+ 314us) server_negotiation.cc:407] Connection header received
0929 20:30:02.505899 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:02.505905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:02.505991 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:02.506097 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:02.506810 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:02.507579 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:02.508780 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:02.509196 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:02.509366 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:02.513102 (+ 3736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.130:51751: BlockingRecv error: recv got EOF from 127.28.244.130:51751 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 29 20:30:02 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.131@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:02 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.131@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:02.973557 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:02.965252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.131:33647)
0929 20:30:02.965785 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:02.965805 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:02.965918 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:02.966218 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:02.966221 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:02.966532 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:02.966709 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:02.966717 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:02.967778 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:02.967786 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:02.968880 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:02.968894 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:02.969672 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:02.969678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:02.969859 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:02.970503 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:02.970520 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:02.973300 (+ 2780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":276,"threads_started":1}
W20260929 20:30:02.973752 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:02.965331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:33647 (local address 127.28.244.190:45043)
0929 20:30:02.965520 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:02.965524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:02.965877 (+ 353us) server_negotiation.cc:407] Connection header received
0929 20:30:02.966029 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:02.966033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:02.966097 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:02.966174 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:02.966858 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:02.967642 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:02.969035 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:02.969520 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:02.969647 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:02.973519 (+ 3872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.131:33647: BlockingRecv error: recv got EOF from 127.28.244.131:33647 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 29 20:30:03 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.129@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:03 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.129@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:03.060665 6445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:03.052547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.129:56773)
0929 20:30:03.053034 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:03.053049 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:03.053164 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:03.053442 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:03.053445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:03.053806 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:03.053953 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:03.053985 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:03.054913 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:03.054916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:03.055924 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:03.055936 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:03.056675 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:03.056679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:03.056795 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:03.057426 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:03.057488 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:03.060440 (+ 2952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":260,"threads_started":1}
W20260929 20:30:03.060930 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:03.052542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:56773 (local address 127.28.244.190:45043)
0929 20:30:03.052735 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:03.052741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:03.053126 (+ 385us) server_negotiation.cc:407] Connection header received
0929 20:30:03.053262 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:03.053265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:03.053325 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:03.053397 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:03.054097 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:03.054809 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:03.056089 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:03.056546 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:03.056652 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:03.060671 (+ 4019us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.129:56773: BlockingRecv error: recv got EOF from 127.28.244.129:56773 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 20:30:03 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.130@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:03 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.130@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:03.523618 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:03.514396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.130:54711)
0929 20:30:03.515042 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:03.515058 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:03.515237 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:03.515606 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:03.515610 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:03.516084 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:03.516301 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:03.516315 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:03.517792 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:03.517801 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:03.518883 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:03.518895 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:03.519641 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:03.519645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:03.519774 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:03.520538 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:03.520557 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:03.523375 (+ 2818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":399,"threads_started":1}
W20260929 20:30:03.524006 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:03.514509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:54711 (local address 127.28.244.190:45043)
0929 20:30:03.514803 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:03.514811 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:03.515160 (+ 349us) server_negotiation.cc:407] Connection header received
0929 20:30:03.515387 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:03.515392 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:03.515464 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:03.515575 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:03.516495 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:03.517596 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:03.519046 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:03.519490 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:03.519588 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:03.523616 (+ 4028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.130:54711: BlockingRecv error: recv got EOF from 127.28.244.130:54711 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 29 20:30:03 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.131@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:03 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.131@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:03.983476 6447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:03.974566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.131:52225)
0929 20:30:03.975083 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:03.975103 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:03.975214 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:03.975516 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:03.975519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:03.975895 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:03.976066 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:03.976074 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:03.977191 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:03.977196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:03.978202 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:03.978214 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:03.979121 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:03.979129 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:03.979296 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:03.980076 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:03.980104 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:03.983242 (+ 3138us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":290,"threads_started":1}
W20260929 20:30:03.983780 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:03.974576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:52225 (local address 127.28.244.190:45043)
0929 20:30:03.974762 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:03.974767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:03.975181 (+ 414us) server_negotiation.cc:407] Connection header received
0929 20:30:03.975314 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:03.975318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:03.975387 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:03.975481 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:03.976180 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:03.977029 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:03.978356 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:03.978920 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:03.979059 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:03.983507 (+ 4448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.131:52225: BlockingRecv error: recv got EOF from 127.28.244.131:52225 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 29 20:30:04 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.129@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:04 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.129@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:04.070755 6448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:04.061801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.129:51727)
0929 20:30:04.062373 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:04.062390 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:04.062512 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:04.062903 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:04.062907 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:04.063381 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:04.063567 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:04.063577 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:04.064679 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:04.064684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:04.065749 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:04.065763 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:04.066660 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:04.066664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:04.066868 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:04.067556 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:04.067579 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:04.070505 (+ 2926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":300,"threads_started":1}
W20260929 20:30:04.071072 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:04.062018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:51727 (local address 127.28.244.190:45043)
0929 20:30:04.062233 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:04.062238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:04.062478 (+ 240us) server_negotiation.cc:407] Connection header received
0929 20:30:04.062622 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:04.062626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:04.062690 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:04.062787 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:04.063739 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:04.064507 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:04.065938 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:04.066489 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:04.066644 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:04.070763 (+ 4119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.129:51727: BlockingRecv error: recv got EOF from 127.28.244.129:51727 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 20:30:04 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.130@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:04 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.130@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:04.533622 6449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:04.524797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.130:36701)
0929 20:30:04.525398 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:04.525416 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:04.525532 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:04.525930 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:04.525935 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:04.526363 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:04.526528 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:04.526536 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:04.527680 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:04.527687 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:04.528693 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:04.528705 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:04.529564 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:04.529571 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:04.529789 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:04.530576 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:04.530595 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:04.533364 (+ 2769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":319,"threads_started":1}
W20260929 20:30:04.533911 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:04.525042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:36701 (local address 127.28.244.190:45043)
0929 20:30:04.525219 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:04.525224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:04.525514 (+ 290us) server_negotiation.cc:407] Connection header received
0929 20:30:04.525694 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:04.525699 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:04.525771 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:04.525892 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:04.526683 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:04.527530 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:04.528840 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:04.529385 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:04.529531 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:04.533597 (+ 4066us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.130:36701: BlockingRecv error: recv got EOF from 127.28.244.130:36701 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 20:30:04 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.131@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:04 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.131@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:04.993633 6450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:04.984389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.131:59071)
0929 20:30:04.984867 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:04.984887 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:04.985012 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:04.985298 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:04.985301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:04.985680 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:04.985861 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:04.985870 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:04.986949 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:04.986953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:04.988107 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:04.988120 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:04.988936 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:04.988941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:04.989101 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:04.989811 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:04.989837 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:04.993352 (+ 3515us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":240,"threads_started":1}
W20260929 20:30:04.993799 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:04.984543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:59071 (local address 127.28.244.190:45043)
0929 20:30:04.984765 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:04.984772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:04.984972 (+ 200us) server_negotiation.cc:407] Connection header received
0929 20:30:04.985113 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:04.985116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:04.985175 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:04.985258 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:04.986023 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:04.986793 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:04.988262 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:04.988767 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:04.988881 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:04.993579 (+ 4698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.131:59071: BlockingRecv error: recv got EOF from 127.28.244.131:59071 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 29 20:30:05 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.129@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:05 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.129@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:05.080752 6451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:05.071935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.129:41181)
0929 20:30:05.072437 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:05.072454 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:05.072612 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:05.072923 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:05.072926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:05.073296 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:05.073494 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:05.073503 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:05.074632 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:05.074637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:05.075620 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:05.075634 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:05.076558 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:05.076563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:05.076693 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:05.077389 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:05.077415 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:05.080423 (+ 3008us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":276,"threads_started":1}
W20260929 20:30:05.080839 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:05.072010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:41181 (local address 127.28.244.190:45043)
0929 20:30:05.072294 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:05.072301 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:05.072579 (+ 278us) server_negotiation.cc:407] Connection header received
0929 20:30:05.072724 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:05.072728 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:05.072793 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:05.072885 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:05.073662 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:05.074475 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:05.075779 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:05.076378 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:05.076484 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:05.080585 (+ 4101us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.129:41181: BlockingRecv error: recv got EOF from 127.28.244.129:41181 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 29 20:30:05 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.130@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:05 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.130@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:05.543136 6452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:05.534520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.130:37029)
0929 20:30:05.535106 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:05.535126 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:05.535252 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:05.535609 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:05.535612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:05.536034 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:05.536199 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:05.536207 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:05.537295 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:05.537300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:05.538241 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:05.538255 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:05.539036 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:05.539040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:05.539172 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:05.539832 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:05.539852 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:05.542916 (+ 3064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":347,"threads_started":1}
W20260929 20:30:05.543534 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:05.534660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:37029 (local address 127.28.244.190:45043)
0929 20:30:05.534843 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:05.534848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:05.535237 (+ 389us) server_negotiation.cc:407] Connection header received
0929 20:30:05.535357 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:05.535361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:05.535438 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:05.535532 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:05.536360 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:05.537134 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:05.538412 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:05.538857 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:05.538964 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:05.543280 (+ 4316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.130:37029: BlockingRecv error: recv got EOF from 127.28.244.130:37029 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 20:30:05 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.131@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:05 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.131@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:06.003582 6453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:05.994763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.131:38461)
0929 20:30:05.995312 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:05.995332 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:05.995454 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:05.995747 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:05.995750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:05.996159 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:05.996303 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:05.996312 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:05.997493 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:05.997497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:05.998884 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:05.998900 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:05.999701 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:05.999706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:05.999882 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:06.000580 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:06.000598 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:06.003363 (+ 2765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":332,"threads_started":1}
W20260929 20:30:06.003861 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:05.994850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:38461 (local address 127.28.244.190:45043)
0929 20:30:05.995094 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:05.995103 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:05.995473 (+ 370us) server_negotiation.cc:407] Connection header received
0929 20:30:05.995546 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:05.995549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:05.995614 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:05.995705 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:05.996485 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:05.997299 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:05.999058 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:05.999538 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:05.999650 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:06.003580 (+ 3930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.131:38461: BlockingRecv error: recv got EOF from 127.28.244.131:38461 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 20:30:06 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.129@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:06 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.129@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:06.090746 6454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:06.081762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.129:49097)
0929 20:30:06.082392 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:06.082412 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:06.082583 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:06.082967 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:06.082973 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:06.083415 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:06.083583 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:06.083592 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:06.084676 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:06.084683 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:06.085589 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:06.085600 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:06.086381 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:06.086386 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:06.086535 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:06.087191 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:06.087220 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:06.090332 (+ 3112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":291,"threads_started":1}
W20260929 20:30:06.090835 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:06.081903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:49097 (local address 127.28.244.190:45043)
0929 20:30:06.082191 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:06.082200 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:06.082509 (+ 309us) server_negotiation.cc:407] Connection header received
0929 20:30:06.082719 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:06.082727 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:06.082798 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:06.082911 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:06.083735 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:06.084534 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:06.085725 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:06.086198 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:06.086336 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:06.090574 (+ 4238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.129:49097: BlockingRecv error: recv got EOF from 127.28.244.129:49097 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 20:30:06 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.130@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:06 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.130@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:06.553527 6455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:06.544286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.130:44815)
0929 20:30:06.544837 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:06.544853 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:06.544984 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:06.545310 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:06.545313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:06.545729 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:06.545894 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:06.545902 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:06.546998 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:06.547002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:06.547974 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:06.547988 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:06.548809 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:06.548814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:06.548947 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:06.549773 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:06.549798 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:06.553268 (+ 3470us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":260,"threads_started":1}
W20260929 20:30:06.553925 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:06.544342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:44815 (local address 127.28.244.190:45043)
0929 20:30:06.544631 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:06.544641 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:06.544942 (+ 301us) server_negotiation.cc:407] Connection header received
0929 20:30:06.545092 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:06.545096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:06.545179 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:06.545267 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:06.546042 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:06.546850 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:06.548145 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:06.548610 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:06.548733 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:06.553543 (+ 4810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.130:44815: BlockingRecv error: recv got EOF from 127.28.244.130:44815 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
Sep 29 20:30:07 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.131@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:07 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.131@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:07.017093 6456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:07.004618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.131:34195)
0929 20:30:07.005272 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:07.005292 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:07.005413 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:07.005793 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:07.005796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:07.006210 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:07.006370 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:07.006378 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:07.007350 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:07.007354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:07.008335 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:07.008347 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:07.009202 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:07.009207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:07.009328 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:07.013708 (+ 4380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:07.013727 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:07.016719 (+ 2992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":350,"threads_started":1}
W20260929 20:30:07.017166 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:07.004750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:34195 (local address 127.28.244.190:45043)
0929 20:30:07.004929 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:07.004934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:07.005403 (+ 469us) server_negotiation.cc:407] Connection header received
0929 20:30:07.005575 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:07.005579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:07.005655 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:07.005747 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:07.006510 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:07.007230 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:07.008495 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:07.009042 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:07.009177 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:07.016952 (+ 7775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.131:34195: BlockingRecv error: recv got EOF from 127.28.244.131:34195 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 20:30:07 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.129@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:07 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.129@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:07.100848 6457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:07.091794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.129:46611)
0929 20:30:07.092437 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:07.092459 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:07.092596 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:07.092913 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:07.092915 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:07.093303 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:07.093459 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:07.093467 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:07.094524 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:07.094532 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:07.095760 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:07.095777 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:07.096816 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:07.096821 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:07.096960 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:07.097539 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:07.097558 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:07.100512 (+ 2954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":300,"threads_started":1}
W20260929 20:30:07.101099 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:07.092052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:46611 (local address 127.28.244.190:45043)
0929 20:30:07.092258 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:07.092263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:07.092568 (+ 305us) server_negotiation.cc:407] Connection header received
0929 20:30:07.092706 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:07.092710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:07.092777 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:07.092876 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:07.093612 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:07.094339 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:07.095929 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:07.096620 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:07.096726 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:07.100715 (+ 3989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.129:46611: BlockingRecv error: recv got EOF from 127.28.244.129:46611 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 29 20:30:07 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.130@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:07 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.130@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:07.563710 6458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:07.554606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.130:54033)
0929 20:30:07.555127 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:07.555145 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:07.555263 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:07.555569 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:07.555572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:07.556048 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:07.556217 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:07.556226 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:07.557522 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:07.557526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:07.558543 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:07.558560 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:07.559382 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:07.559387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:07.559505 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:07.560139 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:07.560157 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:07.563467 (+ 3310us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":268,"threads_started":1}
W20260929 20:30:07.563970 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:07.554778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:54033 (local address 127.28.244.190:45043)
0929 20:30:07.555037 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:07.555044 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:07.555231 (+ 187us) server_negotiation.cc:407] Connection header received
0929 20:30:07.555364 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:07.555368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:07.555431 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:07.555524 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:07.556344 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:07.557317 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:07.558684 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:07.559224 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:07.559318 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:07.563680 (+ 4362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.130:54033: BlockingRecv error: recv got EOF from 127.28.244.130:54033 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 29 20:30:08 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.131@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:08 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.131@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:08.026729 6459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:08.018173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.131:57563)
0929 20:30:08.018780 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:08.018805 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:08.018958 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:08.019302 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:08.019307 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:08.019738 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:08.019922 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:08.019931 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:08.021044 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:08.021050 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:08.021984 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:08.021995 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:08.022720 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:08.022726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:08.022892 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:08.023513 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:08.023530 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:08.026398 (+ 2868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":502,"threads_started":1}
W20260929 20:30:08.026903 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:08.018256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:57563 (local address 127.28.244.190:45043)
0929 20:30:08.018497 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:08.018504 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:08.018884 (+ 380us) server_negotiation.cc:407] Connection header received
0929 20:30:08.019092 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:08.019097 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:08.019167 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:08.019269 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:08.020074 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:08.020887 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:08.022124 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:08.022534 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:08.022684 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:08.026615 (+ 3931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.131:57563: BlockingRecv error: recv got EOF from 127.28.244.131:57563 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 29 20:30:08 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.129@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:08 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.129@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:08.110941 6460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:08.102029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.129:52999)
0929 20:30:08.102540 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:08.102557 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:08.102686 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:08.103034 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:08.103037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:08.103413 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:08.103591 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:08.103600 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:08.104927 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:08.104932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:08.106144 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:08.106163 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:08.107029 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:08.107034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:08.107155 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:08.107782 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:08.107799 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:08.110703 (+ 2904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":297,"threads_started":1}
W20260929 20:30:08.111218 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:08.102089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:52999 (local address 127.28.244.190:45043)
0929 20:30:08.102364 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:08.102370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:08.102697 (+ 327us) server_negotiation.cc:407] Connection header received
0929 20:30:08.102812 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:08.102816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:08.102883 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:08.102974 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:08.103730 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:08.104784 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:08.106327 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:08.106850 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:08.106964 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:08.110926 (+ 3962us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.129:52999: BlockingRecv error: recv got EOF from 127.28.244.129:52999 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 29 20:30:08 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.130@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:08 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.130@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:08.574018 6461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:08.564818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.130:53161)
0929 20:30:08.565379 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:08.565394 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:08.565572 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:08.565926 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:08.565931 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:08.566373 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:08.566546 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:08.566555 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:08.567593 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:08.567600 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:08.568573 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:08.568584 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:08.569368 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:08.569376 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:08.569568 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:08.570208 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:08.570227 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:08.573715 (+ 3488us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":304,"threads_started":1}
W20260929 20:30:08.574178 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:08.564916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:53161 (local address 127.28.244.190:45043)
0929 20:30:08.565178 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:08.565188 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:08.565490 (+ 302us) server_negotiation.cc:407] Connection header received
0929 20:30:08.565710 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:08.565720 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:08.565785 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:08.565895 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:08.566699 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:08.567440 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:08.568720 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:08.569190 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:08.569333 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:08.573906 (+ 4573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.130:53161: BlockingRecv error: recv got EOF from 127.28.244.130:53161 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 29 20:30:09 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.131@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:09 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.131@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:09.036371 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:09.027763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.131:56685)
0929 20:30:09.028344 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:09.028374 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:09.028492 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:09.028795 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:09.028798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:09.029205 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:09.029370 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:09.029377 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:09.030465 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:09.030469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:09.031433 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:09.031445 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:09.032272 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:09.032277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:09.032427 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:09.033010 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:09.033028 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:09.036113 (+ 3085us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":316,"threads_started":1}
W20260929 20:30:09.036586 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:09.027873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:56685 (local address 127.28.244.190:45043)
0929 20:30:09.028041 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:09.028045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:09.028450 (+ 405us) server_negotiation.cc:407] Connection header received
0929 20:30:09.028593 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:09.028597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:09.028668 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:09.028754 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:09.029529 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:09.030267 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:09.031599 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:09.032120 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:09.032228 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:09.036352 (+ 4124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.131:56685: BlockingRecv error: recv got EOF from 127.28.244.131:56685 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 20:30:09 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.129@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:09 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.129@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:09.121862 6463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:09.112231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.129:35769)
0929 20:30:09.112860 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:09.112881 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:09.113042 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:09.113495 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:09.113501 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:09.114056 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:09.114257 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:09.114272 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:09.115627 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:09.115631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:09.116601 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:09.116614 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:09.117440 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:09.117446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:09.117573 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:09.118260 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:09.118279 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:09.121613 (+ 3334us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":396,"threads_started":1}
W20260929 20:30:09.122110 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:09.112348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:35769 (local address 127.28.244.190:45043)
0929 20:30:09.112544 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:09.112553 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:09.113012 (+ 459us) server_negotiation.cc:407] Connection header received
0929 20:30:09.113171 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:09.113177 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:09.113303 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:09.113452 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:09.114433 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:09.115492 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:09.116756 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:09.117205 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:09.117311 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:09.121826 (+ 4515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.129:35769: BlockingRecv error: recv got EOF from 127.28.244.129:35769 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 20:30:09 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.130@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:09 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.130@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:09.584771 6464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:09.575539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.130:34099)
0929 20:30:09.576252 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:09.576270 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:09.576453 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:09.576878 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:09.576884 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:09.577430 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:09.577650 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:09.577662 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:09.578902 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:09.578914 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:09.579923 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:09.579937 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:09.580709 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:09.580714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:09.580830 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:09.581444 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:09.581464 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:09.584425 (+ 2961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":457,"threads_started":1}
W20260929 20:30:09.584918 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:09.575526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:34099 (local address 127.28.244.190:45043)
0929 20:30:09.575826 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:09.575833 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:09.576405 (+ 572us) server_negotiation.cc:407] Connection header received
0929 20:30:09.576604 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:09.576608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:09.576701 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:09.576816 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:09.577825 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:09.578755 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:09.580077 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:09.580550 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:09.580649 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:09.584655 (+ 4006us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.130:34099: BlockingRecv error: recv got EOF from 127.28.244.130:34099 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 29 20:30:10 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.131@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:10 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.131@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:10.046876 6465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:10.037524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.131:54863)
0929 20:30:10.038042 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:10.038066 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:10.038177 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:10.038474 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:10.038478 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:10.038838 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:10.038981 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:10.038989 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:10.040439 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:10.040452 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:10.041737 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:10.041752 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:10.042534 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:10.042539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:10.042666 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:10.043641 (+ 975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:10.043660 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:10.046638 (+ 2978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":251,"threads_started":1}
W20260929 20:30:10.047185 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:10.037787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:54863 (local address 127.28.244.190:45043)
0929 20:30:10.038154 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:10.038158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:10.038178 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:30:10.038274 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:10.038277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:10.038338 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:10.038425 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:10.039184 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:10.040244 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:10.041892 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:10.042354 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:10.042455 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:10.046860 (+ 4405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.131:54863: BlockingRecv error: recv got EOF from 127.28.244.131:54863 (error 108)
Metrics: {"server-negotiator.queue_time_us":240}
Sep 29 20:30:10 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.129@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:10 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.129@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:10.132161 6466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:10.122981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.129:59037)
0929 20:30:10.123614 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:10.123631 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:10.123758 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:10.124214 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:10.124217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:10.124658 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:10.124833 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:10.124842 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:10.125927 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:10.125931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:10.126866 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:10.126877 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:10.127693 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:10.127699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:10.127923 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:10.128686 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:10.128704 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:10.131766 (+ 3062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":371,"threads_started":1}
W20260929 20:30:10.132241 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:10.123108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:59037 (local address 127.28.244.190:45043)
0929 20:30:10.123494 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:10.123501 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:10.123717 (+ 216us) server_negotiation.cc:407] Connection header received
0929 20:30:10.123979 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:10.123983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:10.124071 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:10.124169 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:10.124985 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:10.125786 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:10.127020 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:10.127503 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:10.127663 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:10.132003 (+ 4340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.129:59037: BlockingRecv error: recv got EOF from 127.28.244.129:59037 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 29 20:30:10 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.130@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:10 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.130@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:10.594067 6467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:10.585765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.130:46283)
0929 20:30:10.586207 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:10.586223 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:10.586374 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:10.586690 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:10.586693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:10.587108 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:10.587276 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:10.587285 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:10.588344 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:10.588348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:10.589241 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:10.589252 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:10.589935 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:10.589941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:10.590139 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:10.590781 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:10.590798 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:10.593710 (+ 2912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":246,"threads_started":1}
W20260929 20:30:10.594228 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:10.585825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:46283 (local address 127.28.244.190:45043)
0929 20:30:10.586054 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:10.586060 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:10.586312 (+ 252us) server_negotiation.cc:407] Connection header received
0929 20:30:10.586475 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:10.586479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:10.586546 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:10.586657 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:10.587399 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:10.588203 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:10.589358 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:10.589776 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:10.589867 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:10.593965 (+ 4098us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.130:46283: BlockingRecv error: recv got EOF from 127.28.244.130:46283 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 20:30:11 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.131@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:11 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.131@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:11.056345 6468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:11.047947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.131:56843)
0929 20:30:11.048448 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:11.048468 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:11.048577 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:11.048867 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:11.048870 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:11.049276 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:11.049450 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:11.049461 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:11.050479 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:11.050483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:11.051541 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:11.051553 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:11.052368 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:11.052372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:11.052611 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:11.053190 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:11.053207 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:11.056099 (+ 2892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":285,"threads_started":1}
W20260929 20:30:11.056631 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:11.048019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:56843 (local address 127.28.244.190:45043)
0929 20:30:11.048218 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:11.048222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:11.048543 (+ 321us) server_negotiation.cc:407] Connection header received
0929 20:30:11.048678 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:11.048682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:11.048744 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:11.048826 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:11.049599 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:11.050352 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:11.051679 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:11.052221 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:11.052330 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:11.056319 (+ 3989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.131:56843: BlockingRecv error: recv got EOF from 127.28.244.131:56843 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 20:30:11 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.129@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:11 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.129@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:11.141968 6469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:11.133491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.129:51881)
0929 20:30:11.134007 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:11.134028 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:11.134181 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:11.134618 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:11.134624 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:11.135011 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:11.135235 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:11.135247 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:11.136328 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:11.136332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:11.137268 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:11.137279 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:11.137991 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:11.137996 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:11.138125 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:11.138796 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:11.138813 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:11.141624 (+ 2811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":396,"threads_started":1}
W20260929 20:30:11.142102 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:11.133468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:51881 (local address 127.28.244.190:45043)
0929 20:30:11.133686 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:11.133693 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:11.134127 (+ 434us) server_negotiation.cc:407] Connection header received
0929 20:30:11.134303 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:11.134308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:11.134436 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:11.134540 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:11.135367 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:11.136195 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:11.137402 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:11.137862 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:11.137956 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:11.141843 (+ 3887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.129:51881: BlockingRecv error: recv got EOF from 127.28.244.129:51881 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 29 20:30:11 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.130@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:11 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.130@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:11.604283 6470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:11.595090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.130:45655)
0929 20:30:11.595649 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:11.595666 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:11.595792 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:11.596233 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:11.596237 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:11.596668 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:11.596838 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:11.596848 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:11.597991 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:11.597996 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:11.598993 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:11.599007 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:11.599951 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:11.599957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:11.600107 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:11.600795 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:11.600814 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:11.603939 (+ 3125us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":340,"threads_started":1}
W20260929 20:30:11.604386 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:11.595248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:45655 (local address 127.28.244.190:45043)
0929 20:30:11.595472 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:11.595477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:11.595784 (+ 307us) server_negotiation.cc:407] Connection header received
0929 20:30:11.595945 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:11.595949 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:11.596047 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:11.596143 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:11.597031 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:11.597805 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:11.599193 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:11.599694 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:11.599834 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:11.604142 (+ 4308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.130:45655: BlockingRecv error: recv got EOF from 127.28.244.130:45655 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 20:30:12 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.131@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:12 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.131@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:12.069300 6471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:12.057445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.131:34061)
0929 20:30:12.057969 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:12.057992 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:12.058110 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:12.058621 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:12.058625 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:12.059110 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:12.059268 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:12.059277 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:12.060640 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:12.060644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:12.061568 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:12.061580 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:12.062301 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:12.062306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:12.062434 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:12.066069 (+ 3635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:12.066090 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:12.069059 (+ 2969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":309,"threads_started":1}
W20260929 20:30:12.069563 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:12.057533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:34061 (local address 127.28.244.190:45043)
0929 20:30:12.057772 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:12.057777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:12.058066 (+ 289us) server_negotiation.cc:407] Connection header received
0929 20:30:12.058268 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:12.058273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:12.058394 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:12.058512 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:12.059431 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:12.060459 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:12.061718 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:12.062166 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:12.062264 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:12.069290 (+ 7026us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.131:34061: BlockingRecv error: recv got EOF from 127.28.244.131:34061 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 20:30:12 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.129@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:12 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.129@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:12.154266 6472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:12.143056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.129:44919)
0929 20:30:12.143574 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:12.143591 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:12.143706 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:12.144072 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:12.144075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:12.144438 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:12.144601 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:12.144609 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:12.145754 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:12.145762 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:12.147542 (+ 1780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:12.147562 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:12.148808 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:12.148818 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:12.149052 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:12.149863 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:12.149909 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:12.153896 (+ 3987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":282,"threads_started":1}
W20260929 20:30:12.154366 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:12.143229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:44919 (local address 127.28.244.190:45043)
0929 20:30:12.143525 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:12.143534 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:12.143669 (+ 135us) server_negotiation.cc:407] Connection header received
0929 20:30:12.143841 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:12.143845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:12.143923 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:12.144015 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:12.144729 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:12.145596 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:12.147743 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:12.148558 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:12.148739 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:12.154124 (+ 5385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.129:44919: BlockingRecv error: recv got EOF from 127.28.244.129:44919 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Sep 29 20:30:12 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.130@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:12 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.130@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:12.614095 6473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:12.605415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.130:48215)
0929 20:30:12.606020 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:12.606036 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:12.606149 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:12.606503 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:12.606507 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:12.606884 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:12.607050 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:12.607059 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:12.608150 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:12.608154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:12.609150 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:12.609162 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:12.609899 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:12.609904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:12.610038 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:12.610697 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:12.610714 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:12.613832 (+ 3118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":322,"threads_started":1}
W20260929 20:30:12.614307 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:12.605574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:48215 (local address 127.28.244.190:45043)
0929 20:30:12.605759 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:12.605764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:12.606217 (+ 453us) server_negotiation.cc:407] Connection header received
0929 20:30:12.606277 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:12.606281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:12.606347 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:12.606448 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:12.607209 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:12.608019 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:12.609297 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:12.609726 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:12.609822 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:12.614063 (+ 4241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.130:48215: BlockingRecv error: recv got EOF from 127.28.244.130:48215 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 20:30:13 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.131@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:13 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.131@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:13.079684 6474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:13.070412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.131:49641)
0929 20:30:13.071081 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:13.071172 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:13.071347 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:13.071739 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:13.071744 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:13.072231 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:13.072419 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:13.072428 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:13.073587 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:13.073594 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:13.074628 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:13.074641 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:13.075471 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:13.075479 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:13.075664 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:13.076366 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:13.076385 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:13.079281 (+ 2896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":302,"threads_started":1}
W20260929 20:30:13.079715 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:13.070490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:49641 (local address 127.28.244.190:45043)
0929 20:30:13.070729 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:13.070743 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:13.071272 (+ 529us) server_negotiation.cc:407] Connection header received
0929 20:30:13.071491 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:13.071499 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:13.071589 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:13.071704 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:13.072600 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:13.073433 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:13.074792 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:13.075256 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:13.075398 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:13.079492 (+ 4094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.131:49641: BlockingRecv error: recv got EOF from 127.28.244.131:49641 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 29 20:30:13 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.129@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:13 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.129@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:13.164417 6475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:13.155258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.129:41321)
0929 20:30:13.155721 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:13.155738 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:13.155918 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:13.156255 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:13.156259 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:13.156705 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:13.156935 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:13.156946 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:13.158541 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:13.158550 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:13.159583 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:13.159596 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:13.160322 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:13.160326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:13.160443 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:13.161196 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:13.161213 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:13.164070 (+ 2857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":269,"threads_started":1}
W20260929 20:30:13.164558 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:13.155410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:41321 (local address 127.28.244.190:45043)
0929 20:30:13.155716 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:13.155725 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:13.155829 (+ 104us) server_negotiation.cc:407] Connection header received
0929 20:30:13.156039 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:13.156043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:13.156122 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:13.156208 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:13.157095 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:13.158326 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:13.159713 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:13.160184 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:13.160278 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:13.164289 (+ 4011us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.129:41321: BlockingRecv error: recv got EOF from 127.28.244.129:41321 (error 108)
Metrics: {"server-negotiator.queue_time_us":165}
Sep 29 20:30:13 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.130@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:13 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.130@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:13.624315 6476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:13.615357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.130:40831)
0929 20:30:13.615976 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:13.615994 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:13.616116 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:13.616416 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:13.616420 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:13.616783 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:13.617038 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:13.617051 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:13.618469 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:13.618475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:13.619472 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:13.619485 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:13.620344 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:13.620350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:13.620522 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:13.621128 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:13.621145 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:13.623959 (+ 2814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":264,"threads_started":1}
W20260929 20:30:13.624392 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:13.615455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:40831 (local address 127.28.244.190:45043)
0929 20:30:13.615704 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:13.615711 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:13.616100 (+ 389us) server_negotiation.cc:407] Connection header received
0929 20:30:13.616211 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:13.616215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:13.616280 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:13.616375 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:13.617214 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:13.618337 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:13.619621 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:13.620199 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:13.620331 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:13.624183 (+ 3852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.130:40831: BlockingRecv error: recv got EOF from 127.28.244.130:40831 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 20:30:14 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.131@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:14 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.131@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:14.089566 6477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:14.080839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.131:45383)
0929 20:30:14.081338 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:14.081359 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:14.081474 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:14.081845 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:14.081849 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:14.082220 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:14.082405 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:14.082413 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:14.083412 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:14.083416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:14.084380 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:14.084392 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:14.085168 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:14.085173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:14.085309 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:14.086200 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:14.086220 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:14.089309 (+ 3089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":264,"threads_started":1}
W20260929 20:30:14.089962 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:14.080913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:45383 (local address 127.28.244.190:45043)
0929 20:30:14.081129 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:14.081134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:14.081436 (+ 302us) server_negotiation.cc:407] Connection header received
0929 20:30:14.081590 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:14.081594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:14.081657 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:14.081741 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:14.082540 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:14.083267 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:14.084518 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:14.084988 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:14.085096 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:14.089655 (+ 4559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.131:45383: BlockingRecv error: recv got EOF from 127.28.244.131:45383 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 20:30:14 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.129@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:14 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.129@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:14.175261 6478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:14.165806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.129:48823)
0929 20:30:14.166383 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:14.166399 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:14.166517 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:14.166912 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:14.166916 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:14.167368 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:14.167605 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:14.167615 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:14.168964 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:14.168970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:14.170017 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:14.170030 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:14.170887 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:14.170894 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:14.171090 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:14.171680 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:14.171699 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:14.174865 (+ 3166us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":328,"threads_started":1}
W20260929 20:30:14.175401 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:14.165966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:48823 (local address 127.28.244.190:45043)
0929 20:30:14.166239 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:14.166245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:14.166464 (+ 219us) server_negotiation.cc:407] Connection header received
0929 20:30:14.166612 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:14.166616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:14.166704 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:14.166800 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:14.167790 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:14.168797 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:14.170178 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:14.170682 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:14.170858 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:14.175129 (+ 4271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.129:48823: BlockingRecv error: recv got EOF from 127.28.244.129:48823 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 29 20:30:14 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.130@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:30:14 dist-test-slave-p29l krb5kdc[5432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.244.130@KRBTEST.COM for oryx/127.28.244.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:30:14.633960 6479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:14.625642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45043 (local address 127.28.244.130:60161)
0929 20:30:14.626268 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:14.626285 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:14.626442 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:14.626892 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:14.626897 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:14.627315 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:14.627489 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:14.627498 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:14.628552 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:14.628556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:14.629466 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:14.629477 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:14.630170 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:14.630174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:14.630302 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:14.630882 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:14.630899 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:14.633666 (+ 2767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.244.190:45043: Server oryx/127.28.244.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":351,"threads_started":1}
W20260929 20:30:14.634191 6170 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:14.625862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:60161 (local address 127.28.244.190:45043)
0929 20:30:14.626086 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:14.626092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:14.626571 (+ 479us) server_negotiation.cc:407] Connection header received
0929 20:30:14.626635 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:14.626640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:14.626699 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:14.626812 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:14.627632 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:14.628426 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:14.629606 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:14.630042 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:14.630137 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:14.633930 (+ 3793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.244.130:60161: BlockingRecv error: recv got EOF from 127.28.244.130:60161 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 20:30:14.913508 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 6021
I20260929 20:30:14.926543 6481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:14.914634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58420 (local address 127.28.244.129:44727)
0929 20:30:14.915274 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:14.915282 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:14.915309 (+ 27us) server_negotiation.cc:407] Connection header received
0929 20:30:14.915370 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:14.915377 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:14.915640 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:14.915796 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:14.917375 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:14.918459 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:14.919723 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:14.920437 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:14.920556 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:14.923205 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:30:14.923214 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:30:14.923229 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:30:14.923263 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:30:14.925972 (+ 2709us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.244.129@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 20:30:14.926261 (+ 289us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58420: Request ticket server kudu/127.28.244.129@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":326,"threads_started":1}
W20260929 20:30:14.926659 6481 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58420: Request ticket server kudu/127.28.244.129@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 20:30:14.926713 29650 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 6021: Not authorized: Client connection negotiation failed: client connection to 127.28.244.129:44727: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.244.129@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 20:30:14.926959 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 6021
I20260929 20:30:14.944377 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 6172
I20260929 20:30:14.958506 6482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:14.945537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46488 (local address 127.28.244.130:43797)
0929 20:30:14.946214 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:14.946226 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:14.946258 (+ 32us) server_negotiation.cc:407] Connection header received
0929 20:30:14.946323 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:14.946330 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:14.946589 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:14.946771 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:14.948157 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:14.949270 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:14.950456 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:14.951110 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:14.951212 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:14.954376 (+ 3164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:30:14.954384 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:30:14.954400 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:30:14.954439 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:30:14.957824 (+ 3385us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.244.130@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 20:30:14.958163 (+ 339us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46488: Request ticket server kudu/127.28.244.130@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":342,"threads_started":1}
W20260929 20:30:14.958633 6482 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46488: Request ticket server kudu/127.28.244.130@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 20:30:14.958755 29650 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 6172: Not authorized: Client connection negotiation failed: client connection to 127.28.244.130:43797: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.244.130@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 20:30:14.958913 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 6172
I20260929 20:30:14.977353 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 6303
I20260929 20:30:14.990141 6483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:14.978436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47014 (local address 127.28.244.131:36833)
0929 20:30:14.979077 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:14.979086 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:14.979114 (+ 28us) server_negotiation.cc:407] Connection header received
0929 20:30:14.979180 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:14.979187 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:14.979406 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:14.979583 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:14.981029 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:14.982122 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:14.983371 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:14.984047 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:14.984151 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:14.986807 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:30:14.986816 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:30:14.986832 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:30:14.986880 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:30:14.989612 (+ 2732us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.244.131@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 20:30:14.989903 (+ 291us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47014: Request ticket server kudu/127.28.244.131@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":331,"threads_started":1}
W20260929 20:30:14.990273 6483 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47014: Request ticket server kudu/127.28.244.131@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 20:30:14.990335 29650 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 6303: Not authorized: Client connection negotiation failed: client connection to 127.28.244.131:36833: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.244.131@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 20:30:14.990566 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 6303
I20260929 20:30:15.010303 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 5952
I20260929 20:30:15.011751 6014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:30:15.136061 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 5952
2026-09-29T20:30:15Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21264 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T20:30:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:30:15Z Disabled control of system clock
I20260929 20:30:15.220669 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:45589
--webserver_interface=127.28.244.190
--webserver_port=0
--builtin_ntp_servers=127.28.244.148:39807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:45589
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 20:30:15.569463 6488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:30:15.569808 6488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:30:15.569948 6488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:30:15.577615 6488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:30:15.577741 6488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:30:15.577814 6488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:30:15.577904 6488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:30:15.577996 6488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:30:15.589287 6488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:39807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:45589
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:45589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:30:15.590826 6488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:30:15.593035 6488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:30:15.606868 6496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:30:15.606961 6493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:30:15.606935 6494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:30:15.607364 6488 server_base.cc:1034] running on GCE node
I20260929 20:30:15.608589 6488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:30:15.610162 6488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:30:15.611447 6488 hybrid_clock.cc:648] HybridClock initialized: now 1790713815611376 us; error 53 us; skew 500 ppm
I20260929 20:30:15.611960 6488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:30:15.614601 6488 webserver.cc:460] Webserver started at http://127.28.244.190:40423/ using document root <none> and password file <none>
I20260929 20:30:15.615327 6488 fs_manager.cc:359] Metadata directory not provided
I20260929 20:30:15.615454 6488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:30:15.615855 6488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:30:15.618938 6488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "25f8c008b99f4a2f95b6384a8b371a31"
format_stamp: "Formatted at 2026-09-29 20:30:15 on dist-test-slave-p29l"
server_key: "5470352a024e41cd5c3724c4b1449bbb"
server_key_iv: "e62fa80399b8e3306636d8c3fd117453"
server_key_version: "encryptionkey@0"
I20260929 20:30:15.619796 6488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "25f8c008b99f4a2f95b6384a8b371a31"
format_stamp: "Formatted at 2026-09-29 20:30:15 on dist-test-slave-p29l"
server_key: "5470352a024e41cd5c3724c4b1449bbb"
server_key_iv: "e62fa80399b8e3306636d8c3fd117453"
server_key_version: "encryptionkey@0"
I20260929 20:30:15.625700 6488 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:30:15.629945 6502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:30:15.631542 6488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:30:15.631786 6488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "25f8c008b99f4a2f95b6384a8b371a31"
format_stamp: "Formatted at 2026-09-29 20:30:15 on dist-test-slave-p29l"
server_key: "5470352a024e41cd5c3724c4b1449bbb"
server_key_iv: "e62fa80399b8e3306636d8c3fd117453"
server_key_version: "encryptionkey@0"
I20260929 20:30:15.632115 6488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:30:15.644785 6488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:30:15.645797 6488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:30:15.646150 6488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:30:15.666594 6488 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:45589
I20260929 20:30:15.666640 6553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:45589 every 8 connection(s)
I20260929 20:30:15.668151 6488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:30:15.672240 6554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:30:15.674757 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 6488
I20260929 20:30:15.674983 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:30:15.675643 29650 external_mini_cluster.cc:1428] Setting key 7e5a1f0028646be7761d0eee9b6eb191
I20260929 20:30:15.686425 6554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25f8c008b99f4a2f95b6384a8b371a31: Bootstrap starting.
I20260929 20:30:15.689980 6554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25f8c008b99f4a2f95b6384a8b371a31: Neither blocks nor log segments found. Creating new log.
I20260929 20:30:15.691287 6554 log.cc:824] T 00000000000000000000000000000000 P 25f8c008b99f4a2f95b6384a8b371a31: Log is configured to *not* fsync() on all Append() calls
I20260929 20:30:15.691778 6557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:15.678078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57740 (local address 127.28.244.190:45589)
0929 20:30:15.679197 (+ 1119us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:15.679217 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:15.679263 (+ 46us) server_negotiation.cc:407] Connection header received
0929 20:30:15.681183 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:15.681223 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:15.682131 (+ 908us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:15.682722 (+ 591us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:15.683564 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:15.685367 (+ 1803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:15.686911 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:15.687663 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:15.687921 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:15.689250 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:30:15.689259 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:30:15.689276 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:30:15.689846 (+ 570us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:30:15.689937 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:30:15.689960 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:30:15.690111 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 20:30:15.690314 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 20:30:15.690691 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":409,"threads_started":1}
I20260929 20:30:15.694702 6554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25f8c008b99f4a2f95b6384a8b371a31: No bootstrap required, opened a new log
I20260929 20:30:15.702513 6554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25f8c008b99f4a2f95b6384a8b371a31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f8c008b99f4a2f95b6384a8b371a31" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 45589 } }
I20260929 20:30:15.702978 6554 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25f8c008b99f4a2f95b6384a8b371a31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:30:15.703107 6554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25f8c008b99f4a2f95b6384a8b371a31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25f8c008b99f4a2f95b6384a8b371a31, State: Initialized, Role: FOLLOWER
I20260929 20:30:15.703724 6554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25f8c008b99f4a2f95b6384a8b371a31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f8c008b99f4a2f95b6384a8b371a31" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 45589 } }
I20260929 20:30:15.704061 6554 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25f8c008b99f4a2f95b6384a8b371a31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:30:15.704221 6554 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25f8c008b99f4a2f95b6384a8b371a31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:30:15.704419 6554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25f8c008b99f4a2f95b6384a8b371a31 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:30:15.707288 6554 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25f8c008b99f4a2f95b6384a8b371a31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f8c008b99f4a2f95b6384a8b371a31" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 45589 } }
I20260929 20:30:15.707963 6554 leader_election.cc:302] T 00000000000000000000000000000000 P 25f8c008b99f4a2f95b6384a8b371a31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25f8c008b99f4a2f95b6384a8b371a31; no voters:
I20260929 20:30:15.708598 6554 leader_election.cc:288] T 00000000000000000000000000000000 P 25f8c008b99f4a2f95b6384a8b371a31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:30:15.708768 6559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25f8c008b99f4a2f95b6384a8b371a31 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:30:15.709522 6559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25f8c008b99f4a2f95b6384a8b371a31 [term 1 LEADER]: Becoming Leader. State: Replica: 25f8c008b99f4a2f95b6384a8b371a31, State: Running, Role: LEADER
I20260929 20:30:15.710104 6559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25f8c008b99f4a2f95b6384a8b371a31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f8c008b99f4a2f95b6384a8b371a31" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 45589 } }
I20260929 20:30:15.710848 6554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25f8c008b99f4a2f95b6384a8b371a31 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:30:15.713171 6560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25f8c008b99f4a2f95b6384a8b371a31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25f8c008b99f4a2f95b6384a8b371a31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f8c008b99f4a2f95b6384a8b371a31" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 45589 } } }
I20260929 20:30:15.714030 6560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25f8c008b99f4a2f95b6384a8b371a31 [sys.catalog]: This master's current role is: LEADER
I20260929 20:30:15.714030 6561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25f8c008b99f4a2f95b6384a8b371a31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25f8c008b99f4a2f95b6384a8b371a31. Latest consensus state: current_term: 1 leader_uuid: "25f8c008b99f4a2f95b6384a8b371a31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f8c008b99f4a2f95b6384a8b371a31" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 45589 } } }
I20260929 20:30:15.714497 6561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25f8c008b99f4a2f95b6384a8b371a31 [sys.catalog]: This master's current role is: LEADER
I20260929 20:30:15.715518 6566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:30:15.721706 6566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:30:15.732959 6566 catalog_manager.cc:1347] Generated new cluster ID: be513819c10448b391c36d91f9aeef20
I20260929 20:30:15.733210 6566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:30:15.743029 6566 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:30:15.745038 6566 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:30:15.757651 6566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25f8c008b99f4a2f95b6384a8b371a31: Generated new TSK 0
I20260929 20:30:15.758913 6566 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:30:15.766279 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.129:0
--local_ip_for_outbound_sockets=127.28.244.129
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:45589
--builtin_ntp_servers=127.28.244.148:39807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 20:30:16.124862 6578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:30:16.125219 6578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:30:16.125382 6578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:30:16.132879 6578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:30:16.133044 6578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:30:16.133246 6578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.129
I20260929 20:30:16.142930 6578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:39807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:45589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.129
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:30:16.144418 6578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:30:16.146263 6578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:30:16.160959 6583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:30:16.161003 6586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:30:16.161042 6584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:30:16.162067 6578 server_base.cc:1034] running on GCE node
I20260929 20:30:16.162927 6578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:30:16.164106 6578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:30:16.165347 6578 hybrid_clock.cc:648] HybridClock initialized: now 1790713816165299 us; error 65 us; skew 500 ppm
I20260929 20:30:16.165740 6578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:30:16.168269 6578 webserver.cc:460] Webserver started at http://127.28.244.129:42615/ using document root <none> and password file <none>
I20260929 20:30:16.169018 6578 fs_manager.cc:359] Metadata directory not provided
I20260929 20:30:16.169154 6578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:30:16.169539 6578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:30:16.172549 6578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/data/instance:
uuid: "d6161ba26b634bb19cefaeb7bd3a4ab4"
format_stamp: "Formatted at 2026-09-29 20:30:16 on dist-test-slave-p29l"
server_key: "164725f1e045f4055aceecfbe4cbc25c"
server_key_iv: "afb17456c68ec5434738c2fcdcc36d92"
server_key_version: "encryptionkey@0"
I20260929 20:30:16.173373 6578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance:
uuid: "d6161ba26b634bb19cefaeb7bd3a4ab4"
format_stamp: "Formatted at 2026-09-29 20:30:16 on dist-test-slave-p29l"
server_key: "164725f1e045f4055aceecfbe4cbc25c"
server_key_iv: "afb17456c68ec5434738c2fcdcc36d92"
server_key_version: "encryptionkey@0"
I20260929 20:30:16.178822 6578 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 20:30:16.182668 6592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:30:16.184336 6578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:30:16.184546 6578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/wal
uuid: "d6161ba26b634bb19cefaeb7bd3a4ab4"
format_stamp: "Formatted at 2026-09-29 20:30:16 on dist-test-slave-p29l"
server_key: "164725f1e045f4055aceecfbe4cbc25c"
server_key_iv: "afb17456c68ec5434738c2fcdcc36d92"
server_key_version: "encryptionkey@0"
I20260929 20:30:16.184791 6578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:30:16.205075 6578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:30:16.206032 6578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:30:16.206351 6578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:30:16.207393 6578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:30:16.209045 6578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:30:16.209183 6578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:30:16.209313 6578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:30:16.209405 6578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:30:16.244040 6578 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.129:40299
I20260929 20:30:16.244131 6704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.129:40299 every 8 connection(s)
I20260929 20:30:16.245434 6578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
I20260929 20:30:16.245795 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 6578
I20260929 20:30:16.245991 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance
I20260929 20:30:16.246930 29650 external_mini_cluster.cc:1428] Setting key 3c6d0fdbca6fde2f70e4c6d1cee1e876
I20260929 20:30:16.250301 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.130:0
--local_ip_for_outbound_sockets=127.28.244.130
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:45589
--builtin_ntp_servers=127.28.244.148:39807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 20:30:16.258833 6707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:16.248186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:47675 (local address 127.28.244.190:45589)
0929 20:30:16.248639 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:16.248643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:16.250156 (+ 1513us) server_negotiation.cc:407] Connection header received
0929 20:30:16.252062 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:16.252067 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:16.252161 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:16.252279 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:16.253405 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:16.254126 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:16.255733 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:16.256212 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:16.256346 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:16.257835 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:30:16.257839 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:30:16.257850 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:30:16.257875 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:30:16.257922 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:30:16.257937 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:30:16.258039 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 20:30:16.258356 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 20:30:16.258462 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":242,"threads_started":1}
I20260929 20:30:16.259959 6708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:16.249013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45589 (local address 127.28.244.129:47675)
0929 20:30:16.250008 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:16.250045 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:16.251707 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:16.252542 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:16.252552 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:16.252643 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:16.253225 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:16.253242 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:16.254280 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:16.254290 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:16.255566 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:16.255583 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:16.256453 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:16.256461 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:16.256708 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:16.257464 (+ 756us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:30:16.257467 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:16.257512 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:30:16.257598 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:30:16.258038 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:30:16.258051 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:30:16.258061 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 20:30:16.258317 (+ 256us) client_negotiation.cc:240] Negotiation successful
0929 20:30:16.258756 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":344,"threads_started":1}
I20260929 20:30:16.261668 6705 heartbeater.cc:347] Connected to a master server at 127.28.244.190:45589
I20260929 20:30:16.261997 6705 heartbeater.cc:464] Registering TS with master...
I20260929 20:30:16.262816 6705 heartbeater.cc:511] Master 127.28.244.190:45589 requested a full tablet report, sending...
I20260929 20:30:16.265242 6519 ts_manager.cc:194] Registered new tserver with Master: d6161ba26b634bb19cefaeb7bd3a4ab4 (127.28.244.129:40299)
I20260929 20:30:16.267505 6519 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.244.129:47675
W20260929 20:30:16.584870 6709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:30:16.585152 6709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:30:16.585297 6709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:30:16.592758 6709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:30:16.592865 6709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:30:16.593042 6709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.130
I20260929 20:30:16.602236 6709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:39807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:45589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.130
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:30:16.603590 6709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:30:16.605365 6709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:30:16.618611 6714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:30:16.618518 6715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:30:16.618624 6717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:30:16.619370 6709 server_base.cc:1034] running on GCE node
I20260929 20:30:16.620090 6709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:30:16.621089 6709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:30:16.622347 6709 hybrid_clock.cc:648] HybridClock initialized: now 1790713816622279 us; error 55 us; skew 500 ppm
I20260929 20:30:16.622709 6709 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:30:16.625227 6709 webserver.cc:460] Webserver started at http://127.28.244.130:43399/ using document root <none> and password file <none>
I20260929 20:30:16.626019 6709 fs_manager.cc:359] Metadata directory not provided
I20260929 20:30:16.626159 6709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:30:16.626497 6709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:30:16.629537 6709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/data/instance:
uuid: "c5a200b75803406b8e8d690c3f44605d"
format_stamp: "Formatted at 2026-09-29 20:30:16 on dist-test-slave-p29l"
server_key: "ef4dbfe4456a86103852812712caa9c3"
server_key_iv: "c1d666d27769ee88808df9afd89d695c"
server_key_version: "encryptionkey@0"
I20260929 20:30:16.630376 6709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance:
uuid: "c5a200b75803406b8e8d690c3f44605d"
format_stamp: "Formatted at 2026-09-29 20:30:16 on dist-test-slave-p29l"
server_key: "ef4dbfe4456a86103852812712caa9c3"
server_key_iv: "c1d666d27769ee88808df9afd89d695c"
server_key_version: "encryptionkey@0"
I20260929 20:30:16.635903 6709 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260929 20:30:16.639536 6723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:30:16.640936 6709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:30:16.641129 6709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/wal
uuid: "c5a200b75803406b8e8d690c3f44605d"
format_stamp: "Formatted at 2026-09-29 20:30:16 on dist-test-slave-p29l"
server_key: "ef4dbfe4456a86103852812712caa9c3"
server_key_iv: "c1d666d27769ee88808df9afd89d695c"
server_key_version: "encryptionkey@0"
I20260929 20:30:16.641369 6709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:30:16.659505 6709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:30:16.660549 6709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:30:16.660880 6709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:30:16.661921 6709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:30:16.663475 6709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:30:16.663632 6709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:30:16.663739 6709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:30:16.663846 6709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:30:16.700924 6709 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.130:39413
I20260929 20:30:16.700971 6835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.130:39413 every 8 connection(s)
I20260929 20:30:16.702270 6709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
I20260929 20:30:16.704747 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 6709
I20260929 20:30:16.704918 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance
I20260929 20:30:16.705482 29650 external_mini_cluster.cc:1428] Setting key c56795ce6f40ac3a1278ab0d38e083e9
I20260929 20:30:16.708810 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.131:0
--local_ip_for_outbound_sockets=127.28.244.131
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:45589
--builtin_ntp_servers=127.28.244.148:39807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 20:30:16.716351 6707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:16.705089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:53883 (local address 127.28.244.190:45589)
0929 20:30:16.705264 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:16.705269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:16.707065 (+ 1796us) server_negotiation.cc:407] Connection header received
0929 20:30:16.709214 (+ 2149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:16.709220 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:16.709315 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:16.709428 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:16.710594 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:16.711385 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:16.713204 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:16.713626 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:16.713772 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:16.715269 (+ 1497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:30:16.715278 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:30:16.715284 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:30:16.715312 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:30:16.715357 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:30:16.715403 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:30:16.715508 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 20:30:16.715835 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 20:30:16.715964 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 20:30:16.717576 6838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:16.705750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45589 (local address 127.28.244.130:53883)
0929 20:30:16.706876 (+ 1126us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:16.706923 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:16.708845 (+ 1922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:16.709697 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:16.709707 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:16.709806 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:16.710416 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:16.710434 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:16.711549 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:16.711555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:16.713036 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:16.713051 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:16.713870 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:16.713878 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:16.714127 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:16.714902 (+ 775us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:30:16.714904 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:16.714954 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:30:16.715043 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:30:16.715489 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:30:16.715498 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:30:16.715503 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 20:30:16.715772 (+ 269us) client_negotiation.cc:240] Negotiation successful
0929 20:30:16.716198 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":354,"threads_started":1}
I20260929 20:30:16.719215 6836 heartbeater.cc:347] Connected to a master server at 127.28.244.190:45589
I20260929 20:30:16.719611 6836 heartbeater.cc:464] Registering TS with master...
I20260929 20:30:16.720564 6836 heartbeater.cc:511] Master 127.28.244.190:45589 requested a full tablet report, sending...
I20260929 20:30:16.722234 6519 ts_manager.cc:194] Registered new tserver with Master: c5a200b75803406b8e8d690c3f44605d (127.28.244.130:39413)
I20260929 20:30:16.723209 6519 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.244.130:53883
W20260929 20:30:17.062260 6839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:30:17.062583 6839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:30:17.062731 6839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:30:17.070863 6839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:30:17.071022 6839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:30:17.071225 6839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.131
I20260929 20:30:17.081979 6839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:39807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:45589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.131
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:30:17.083528 6839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:30:17.085598 6839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:30:17.102954 6845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:30:17.103127 6844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:30:17.103269 6839 server_base.cc:1034] running on GCE node
W20260929 20:30:17.103003 6847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:30:17.104184 6839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:30:17.105288 6839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:30:17.106575 6839 hybrid_clock.cc:648] HybridClock initialized: now 1790713817106492 us; error 72 us; skew 500 ppm
I20260929 20:30:17.106968 6839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:30:17.114228 6839 webserver.cc:460] Webserver started at http://127.28.244.131:34027/ using document root <none> and password file <none>
I20260929 20:30:17.115151 6839 fs_manager.cc:359] Metadata directory not provided
I20260929 20:30:17.115320 6839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:30:17.115798 6839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:30:17.119242 6839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/data/instance:
uuid: "cebb8e81ab394d889a0660be9313d1bf"
format_stamp: "Formatted at 2026-09-29 20:30:17 on dist-test-slave-p29l"
server_key: "93aaa3771115a64cf09e470a90a2dd7b"
server_key_iv: "aacebc0bdf71cf1d97d806e3492920b2"
server_key_version: "encryptionkey@0"
I20260929 20:30:17.120292 6839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance:
uuid: "cebb8e81ab394d889a0660be9313d1bf"
format_stamp: "Formatted at 2026-09-29 20:30:17 on dist-test-slave-p29l"
server_key: "93aaa3771115a64cf09e470a90a2dd7b"
server_key_iv: "aacebc0bdf71cf1d97d806e3492920b2"
server_key_version: "encryptionkey@0"
I20260929 20:30:17.126240 6839 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:30:17.130486 6853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:30:17.132273 6839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:30:17.132565 6839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/wal
uuid: "cebb8e81ab394d889a0660be9313d1bf"
format_stamp: "Formatted at 2026-09-29 20:30:17 on dist-test-slave-p29l"
server_key: "93aaa3771115a64cf09e470a90a2dd7b"
server_key_iv: "aacebc0bdf71cf1d97d806e3492920b2"
server_key_version: "encryptionkey@0"
I20260929 20:30:17.132844 6839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:30:17.148167 6839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:30:17.149163 6839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:30:17.149497 6839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:30:17.150565 6839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:30:17.152295 6839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:30:17.152451 6839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:30:17.152594 6839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:30:17.152681 6839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:30:17.188133 6839 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.131:38619
I20260929 20:30:17.188150 6965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.131:38619 every 8 connection(s)
I20260929 20:30:17.189695 6839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
I20260929 20:30:17.195240 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 6839
I20260929 20:30:17.195410 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance
I20260929 20:30:17.196002 29650 external_mini_cluster.cc:1428] Setting key b980895d3b3f8c66dab46d20ba88f751
I20260929 20:30:17.203753 6707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:17.192552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:36255 (local address 127.28.244.190:45589)
0929 20:30:17.192753 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:17.192757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:17.194267 (+ 1510us) server_negotiation.cc:407] Connection header received
0929 20:30:17.196784 (+ 2517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:17.196790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:17.196896 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:17.197027 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:17.198369 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:17.199179 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:17.200878 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:17.201306 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:17.201453 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:17.202868 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:30:17.202873 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:30:17.202876 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:30:17.202896 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:30:17.202929 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:30:17.202944 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:30:17.203063 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 20:30:17.203307 (+ 244us) server_negotiation.cc:299] Negotiation successful
0929 20:30:17.203413 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 20:30:17.204903 6968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:17.193123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:45589 (local address 127.28.244.131:36255)
0929 20:30:17.194091 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:17.194137 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:17.196416 (+ 2279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:17.197290 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:17.197300 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:17.197402 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:17.198193 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:17.198214 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:17.199350 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:17.199354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:17.200723 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:17.200737 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:17.201554 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:17.201562 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:17.201809 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:17.202518 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:30:17.202521 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:17.202566 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:30:17.202652 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:30:17.203062 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:30:17.203074 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:30:17.203085 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 20:30:17.203282 (+ 197us) client_negotiation.cc:240] Negotiation successful
0929 20:30:17.203720 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":290,"threads_started":1}
I20260929 20:30:17.206684 6966 heartbeater.cc:347] Connected to a master server at 127.28.244.190:45589
I20260929 20:30:17.207011 6966 heartbeater.cc:464] Registering TS with master...
I20260929 20:30:17.207801 6966 heartbeater.cc:511] Master 127.28.244.190:45589 requested a full tablet report, sending...
I20260929 20:30:17.209679 6519 ts_manager.cc:194] Registered new tserver with Master: cebb8e81ab394d889a0660be9313d1bf (127.28.244.131:38619)
I20260929 20:30:17.210809 6519 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.244.131:36255
I20260929 20:30:17.212491 29650 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 20:30:17.219751 6707 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:17.217811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57742 (local address 127.28.244.190:45589)
0929 20:30:17.218021 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:17.218026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:17.218643 (+ 617us) server_negotiation.cc:407] Connection header received
0929 20:30:17.218722 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:17.218726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:17.218797 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:17.218905 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:17.219406 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57742: BlockingRecv error: recv got EOF from 127.0.0.1:57742 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 20:30:17.220717 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 6578
I20260929 20:30:17.229271 6975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:17.221723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35204 (local address 127.28.244.129:40299)
0929 20:30:17.222308 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:17.222318 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:17.222355 (+ 37us) server_negotiation.cc:407] Connection header received
0929 20:30:17.222416 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:17.222423 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:17.222652 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:17.222823 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:17.223268 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:17.224385 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:17.225712 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:17.226353 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:17.226497 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:17.227669 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:30:17.227677 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:30:17.227693 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:30:17.228390 (+ 697us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:30:17.228439 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:30:17.228460 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:30:17.228602 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 20:30:17.228759 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 20:30:17.228923 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":257,"threads_started":1}
I20260929 20:30:17.230764 6700 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:30:17.313108 6705 heartbeater.cc:503] Master 127.28.244.190:45589 was elected leader, sending a full tablet report...
I20260929 20:30:17.314507 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 6578
W20260929 20:30:17.332211 6506 connection.cc:535] server connection from 127.28.244.129:47675 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 20:30:17.332386 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 6709
I20260929 20:30:17.341650 6977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:17.333434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46102 (local address 127.28.244.130:39413)
0929 20:30:17.334049 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:17.334056 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:17.334086 (+ 30us) server_negotiation.cc:407] Connection header received
0929 20:30:17.334154 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:17.334161 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:17.334388 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:17.334558 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:17.335147 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:17.336513 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:17.337937 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:17.338633 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:17.338739 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:17.340057 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:30:17.340065 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:30:17.340081 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:30:17.340722 (+ 641us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:30:17.340781 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:30:17.340802 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:30:17.340896 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 20:30:17.341096 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 20:30:17.341231 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":349,"threads_started":1}
I20260929 20:30:17.343011 6831 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:30:17.425982 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 6709
I20260929 20:30:17.442968 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 6839
I20260929 20:30:17.452214 6979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:17.444216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38490 (local address 127.28.244.131:38619)
0929 20:30:17.444880 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:17.444910 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:17.444942 (+ 32us) server_negotiation.cc:407] Connection header received
0929 20:30:17.445009 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:17.445016 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:17.445235 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:17.445371 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:17.445885 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:17.447019 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:17.448526 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:17.449183 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:17.449303 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:17.450588 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:30:17.450597 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:30:17.450611 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:30:17.451246 (+ 635us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:30:17.451300 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:30:17.451326 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:30:17.451420 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 20:30:17.451606 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 20:30:17.451780 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":362,"threads_started":1}
I20260929 20:30:17.453677 6961 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:30:17.537746 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 6839
I20260929 20:30:17.554649 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 6488
I20260929 20:30:17.556031 6549 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:30:17.660908 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 6488
2026-09-29T20:30:17Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2520 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T20:30:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:30:17Z Disabled control of system clock
I20260929 20:30:17.741135 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:33039
--webserver_interface=127.28.244.190
--webserver_port=0
--builtin_ntp_servers=127.28.244.148:41169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:33039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 20:30:18.092617 6985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:30:18.093036 6985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:30:18.093204 6985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:30:18.100960 6985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:30:18.101154 6985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:30:18.101253 6985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:30:18.101363 6985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:30:18.101454 6985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:30:18.111269 6985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:41169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:33039
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:33039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:30:18.112870 6985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:30:18.114842 6985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:30:18.129113 6990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:30:18.129424 6991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:30:18.129863 6993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:30:18.130992 6985 server_base.cc:1034] running on GCE node
I20260929 20:30:18.131937 6985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:30:18.133541 6985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:30:18.134783 6985 hybrid_clock.cc:648] HybridClock initialized: now 1790713818134740 us; error 38 us; skew 500 ppm
I20260929 20:30:18.135149 6985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:30:18.137902 6985 webserver.cc:460] Webserver started at http://127.28.244.190:36553/ using document root <none> and password file <none>
I20260929 20:30:18.138679 6985 fs_manager.cc:359] Metadata directory not provided
I20260929 20:30:18.138783 6985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:30:18.139096 6985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:30:18.142468 6985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "e13715020dc94c03b2da79d5e0a6acdf"
format_stamp: "Formatted at 2026-09-29 20:30:18 on dist-test-slave-p29l"
server_key: "dd6d35d8d5c52f3e90229b58536b0d7f"
server_key_iv: "bc886dadb141a5dc6c22aeab22504199"
server_key_version: "encryptionkey@0"
I20260929 20:30:18.143353 6985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "e13715020dc94c03b2da79d5e0a6acdf"
format_stamp: "Formatted at 2026-09-29 20:30:18 on dist-test-slave-p29l"
server_key: "dd6d35d8d5c52f3e90229b58536b0d7f"
server_key_iv: "bc886dadb141a5dc6c22aeab22504199"
server_key_version: "encryptionkey@0"
I20260929 20:30:18.151364 6985 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.009s sys 0.001s
I20260929 20:30:18.156584 6999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:30:18.158234 6985 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 20:30:18.158480 6985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "e13715020dc94c03b2da79d5e0a6acdf"
format_stamp: "Formatted at 2026-09-29 20:30:18 on dist-test-slave-p29l"
server_key: "dd6d35d8d5c52f3e90229b58536b0d7f"
server_key_iv: "bc886dadb141a5dc6c22aeab22504199"
server_key_version: "encryptionkey@0"
I20260929 20:30:18.158741 6985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:30:18.165234 6985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:30:18.166138 6985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:30:18.166486 6985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:30:18.187029 6985 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:33039
I20260929 20:30:18.187090 7050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:33039 every 8 connection(s)
I20260929 20:30:18.188619 6985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:30:18.192936 7051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:30:18.196213 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 6985
I20260929 20:30:18.196451 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:30:18.197237 29650 external_mini_cluster.cc:1428] Setting key f7471ff2ffef0514ba08b17279412755
I20260929 20:30:18.207010 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:18.199676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41968 (local address 127.28.244.190:33039)
0929 20:30:18.200931 (+ 1255us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:18.200944 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:18.200999 (+ 55us) server_negotiation.cc:407] Connection header received
0929 20:30:18.202829 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:18.202858 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:18.203391 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:18.203945 (+ 554us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:18.205201 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:30:18.205208 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:30:18.205224 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:30:18.205239 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:30:18.205307 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:30:18.205324 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:30:18.205586 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 20:30:18.205739 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 20:30:18.206023 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":395,"threads_started":1}
I20260929 20:30:18.207190 7051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e13715020dc94c03b2da79d5e0a6acdf: Bootstrap starting.
I20260929 20:30:18.211140 7051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e13715020dc94c03b2da79d5e0a6acdf: Neither blocks nor log segments found. Creating new log.
I20260929 20:30:18.212348 7051 log.cc:824] T 00000000000000000000000000000000 P e13715020dc94c03b2da79d5e0a6acdf: Log is configured to *not* fsync() on all Append() calls
I20260929 20:30:18.215534 7051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e13715020dc94c03b2da79d5e0a6acdf: No bootstrap required, opened a new log
I20260929 20:30:18.223501 7051 raft_consensus.cc:348] T 00000000000000000000000000000000 P e13715020dc94c03b2da79d5e0a6acdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13715020dc94c03b2da79d5e0a6acdf" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 33039 } }
I20260929 20:30:18.224107 7051 raft_consensus.cc:374] T 00000000000000000000000000000000 P e13715020dc94c03b2da79d5e0a6acdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:30:18.224226 7051 raft_consensus.cc:729] T 00000000000000000000000000000000 P e13715020dc94c03b2da79d5e0a6acdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e13715020dc94c03b2da79d5e0a6acdf, State: Initialized, Role: FOLLOWER
I20260929 20:30:18.224843 7051 consensus_queue.cc:260] T 00000000000000000000000000000000 P e13715020dc94c03b2da79d5e0a6acdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13715020dc94c03b2da79d5e0a6acdf" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 33039 } }
I20260929 20:30:18.225106 7051 raft_consensus.cc:388] T 00000000000000000000000000000000 P e13715020dc94c03b2da79d5e0a6acdf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:30:18.225260 7051 raft_consensus.cc:482] T 00000000000000000000000000000000 P e13715020dc94c03b2da79d5e0a6acdf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:30:18.225442 7051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e13715020dc94c03b2da79d5e0a6acdf [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:30:18.227663 7051 raft_consensus.cc:504] T 00000000000000000000000000000000 P e13715020dc94c03b2da79d5e0a6acdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13715020dc94c03b2da79d5e0a6acdf" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 33039 } }
I20260929 20:30:18.228276 7051 leader_election.cc:302] T 00000000000000000000000000000000 P e13715020dc94c03b2da79d5e0a6acdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e13715020dc94c03b2da79d5e0a6acdf; no voters:
I20260929 20:30:18.229038 7051 leader_election.cc:288] T 00000000000000000000000000000000 P e13715020dc94c03b2da79d5e0a6acdf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:30:18.229197 7056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e13715020dc94c03b2da79d5e0a6acdf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:30:18.230216 7056 raft_consensus.cc:686] T 00000000000000000000000000000000 P e13715020dc94c03b2da79d5e0a6acdf [term 1 LEADER]: Becoming Leader. State: Replica: e13715020dc94c03b2da79d5e0a6acdf, State: Running, Role: LEADER
I20260929 20:30:18.230806 7056 consensus_queue.cc:237] T 00000000000000000000000000000000 P e13715020dc94c03b2da79d5e0a6acdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13715020dc94c03b2da79d5e0a6acdf" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 33039 } }
I20260929 20:30:18.231567 7051 sys_catalog.cc:564] T 00000000000000000000000000000000 P e13715020dc94c03b2da79d5e0a6acdf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:30:18.234004 7058 sys_catalog.cc:455] T 00000000000000000000000000000000 P e13715020dc94c03b2da79d5e0a6acdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader e13715020dc94c03b2da79d5e0a6acdf. Latest consensus state: current_term: 1 leader_uuid: "e13715020dc94c03b2da79d5e0a6acdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13715020dc94c03b2da79d5e0a6acdf" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 33039 } } }
I20260929 20:30:18.234503 7058 sys_catalog.cc:458] T 00000000000000000000000000000000 P e13715020dc94c03b2da79d5e0a6acdf [sys.catalog]: This master's current role is: LEADER
I20260929 20:30:18.236575 7057 sys_catalog.cc:455] T 00000000000000000000000000000000 P e13715020dc94c03b2da79d5e0a6acdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e13715020dc94c03b2da79d5e0a6acdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13715020dc94c03b2da79d5e0a6acdf" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 33039 } } }
I20260929 20:30:18.237123 7057 sys_catalog.cc:458] T 00000000000000000000000000000000 P e13715020dc94c03b2da79d5e0a6acdf [sys.catalog]: This master's current role is: LEADER
I20260929 20:30:18.237954 7064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:30:18.243117 7064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:30:18.253454 7064 catalog_manager.cc:1347] Generated new cluster ID: 73f82ab16d314fda9a1e43cdafc67df8
I20260929 20:30:18.253598 7064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:30:18.273419 7064 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:30:18.273913 7064 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:30:18.284545 7064 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e13715020dc94c03b2da79d5e0a6acdf: Generated new TSK 0
I20260929 20:30:18.285516 7064 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:30:18.299450 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.129:0
--local_ip_for_outbound_sockets=127.28.244.129
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:33039
--builtin_ntp_servers=127.28.244.148:41169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 20:30:18.673229 7075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:30:18.673626 7075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:30:18.673795 7075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:30:18.681666 7075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:30:18.681847 7075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:30:18.682040 7075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.129
I20260929 20:30:18.692478 7075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:41169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:33039
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.129
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:30:18.693955 7075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:30:18.696039 7075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:30:18.711637 7080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:30:18.711731 7083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:30:18.711664 7081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:30:18.713428 7075 server_base.cc:1034] running on GCE node
I20260929 20:30:18.714320 7075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:30:18.715626 7075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:30:18.716940 7075 hybrid_clock.cc:648] HybridClock initialized: now 1790713818716859 us; error 50 us; skew 500 ppm
I20260929 20:30:18.717334 7075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:30:18.720459 7075 webserver.cc:460] Webserver started at http://127.28.244.129:40125/ using document root <none> and password file <none>
I20260929 20:30:18.721292 7075 fs_manager.cc:359] Metadata directory not provided
I20260929 20:30:18.721442 7075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:30:18.721809 7075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:30:18.727290 7075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/data/instance:
uuid: "450232fee9f64c539310bc917a70eb1d"
format_stamp: "Formatted at 2026-09-29 20:30:18 on dist-test-slave-p29l"
server_key: "3809d2c899bc65a811da20a692c61ff5"
server_key_iv: "808f6d888824c39cffb120072523d921"
server_key_version: "encryptionkey@0"
I20260929 20:30:18.728335 7075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance:
uuid: "450232fee9f64c539310bc917a70eb1d"
format_stamp: "Formatted at 2026-09-29 20:30:18 on dist-test-slave-p29l"
server_key: "3809d2c899bc65a811da20a692c61ff5"
server_key_iv: "808f6d888824c39cffb120072523d921"
server_key_version: "encryptionkey@0"
I20260929 20:30:18.734205 7075 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:30:18.738194 7089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:30:18.739961 7075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:30:18.740201 7075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/wal
uuid: "450232fee9f64c539310bc917a70eb1d"
format_stamp: "Formatted at 2026-09-29 20:30:18 on dist-test-slave-p29l"
server_key: "3809d2c899bc65a811da20a692c61ff5"
server_key_iv: "808f6d888824c39cffb120072523d921"
server_key_version: "encryptionkey@0"
I20260929 20:30:18.740465 7075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:30:18.747344 7075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:30:18.748381 7075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:30:18.748685 7075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:30:18.749706 7075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:30:18.751291 7075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:30:18.751425 7075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:30:18.751559 7075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:30:18.751646 7075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:30:18.786441 7075 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.129:38917
I20260929 20:30:18.786486 7201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.129:38917 every 8 connection(s)
I20260929 20:30:18.788065 7075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
I20260929 20:30:18.797380 7204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:18.791152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:38987 (local address 127.28.244.190:33039)
0929 20:30:18.791655 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:18.791662 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:18.793102 (+ 1440us) server_negotiation.cc:407] Connection header received
0929 20:30:18.794831 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:18.794836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:18.794896 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:18.795004 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:18.796577 (+ 1573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:30:18.796582 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:30:18.796585 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:30:18.796603 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:30:18.796647 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:30:18.796659 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:30:18.796782 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 20:30:18.796941 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 20:30:18.796995 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":228,"threads_started":1}
I20260929 20:30:18.797737 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 7075
I20260929 20:30:18.797951 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance
I20260929 20:30:18.798439 7205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:18.791909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:33039 (local address 127.28.244.129:38987)
0929 20:30:18.792919 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:18.792964 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:18.794547 (+ 1583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:18.795288 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:18.795297 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:18.795404 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:18.796243 (+ 839us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:30:18.796246 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:18.796288 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:30:18.796386 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:30:18.796752 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:30:18.796759 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:30:18.796765 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 20:30:18.796992 (+ 227us) client_negotiation.cc:240] Negotiation successful
0929 20:30:18.797235 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":296,"threads_started":1}
I20260929 20:30:18.798646 29650 external_mini_cluster.cc:1428] Setting key 1223f8e2b3964f823bf00a8cb8ec35df
I20260929 20:30:18.800223 7202 heartbeater.cc:347] Connected to a master server at 127.28.244.190:33039
I20260929 20:30:18.800568 7202 heartbeater.cc:464] Registering TS with master...
I20260929 20:30:18.801208 7202 heartbeater.cc:511] Master 127.28.244.190:33039 requested a full tablet report, sending...
I20260929 20:30:18.801555 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.130:0
--local_ip_for_outbound_sockets=127.28.244.130
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:33039
--builtin_ntp_servers=127.28.244.148:41169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 20:30:18.803465 7016 ts_manager.cc:194] Registered new tserver with Master: 450232fee9f64c539310bc917a70eb1d (127.28.244.129:38917)
W20260929 20:30:19.147091 7206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:30:19.147393 7206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:30:19.147492 7206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:30:19.155022 7206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:30:19.155130 7206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:30:19.155241 7206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.130
I20260929 20:30:19.165047 7206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:41169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:33039
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.130
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:30:19.166340 7206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:30:19.168190 7206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:30:19.181708 7211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:30:19.181777 7214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:30:19.181754 7212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:30:19.182675 7206 server_base.cc:1034] running on GCE node
I20260929 20:30:19.183347 7206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:30:19.184587 7206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:30:19.185830 7206 hybrid_clock.cc:648] HybridClock initialized: now 1790713819185793 us; error 60 us; skew 500 ppm
I20260929 20:30:19.186183 7206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:30:19.188864 7206 webserver.cc:460] Webserver started at http://127.28.244.130:43729/ using document root <none> and password file <none>
I20260929 20:30:19.189611 7206 fs_manager.cc:359] Metadata directory not provided
I20260929 20:30:19.189745 7206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:30:19.190075 7206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:30:19.193105 7206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/data/instance:
uuid: "7bb1a641a2164be2b79356b1b3d4ca84"
format_stamp: "Formatted at 2026-09-29 20:30:19 on dist-test-slave-p29l"
server_key: "b38f6bf56dcf7297edabc95063c7d138"
server_key_iv: "fa051032e39a0b932d9b7f655bf54960"
server_key_version: "encryptionkey@0"
I20260929 20:30:19.193915 7206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance:
uuid: "7bb1a641a2164be2b79356b1b3d4ca84"
format_stamp: "Formatted at 2026-09-29 20:30:19 on dist-test-slave-p29l"
server_key: "b38f6bf56dcf7297edabc95063c7d138"
server_key_iv: "fa051032e39a0b932d9b7f655bf54960"
server_key_version: "encryptionkey@0"
I20260929 20:30:19.199978 7206 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 20:30:19.203567 7220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:30:19.205067 7206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:30:19.205268 7206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/wal
uuid: "7bb1a641a2164be2b79356b1b3d4ca84"
format_stamp: "Formatted at 2026-09-29 20:30:19 on dist-test-slave-p29l"
server_key: "b38f6bf56dcf7297edabc95063c7d138"
server_key_iv: "fa051032e39a0b932d9b7f655bf54960"
server_key_version: "encryptionkey@0"
I20260929 20:30:19.205507 7206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:30:19.211611 7206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:30:19.212584 7206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:30:19.212875 7206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:30:19.213873 7206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:30:19.215378 7206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:30:19.215503 7206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:30:19.215638 7206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:30:19.215744 7206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:30:19.251745 7206 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.130:42687
I20260929 20:30:19.251860 7332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.130:42687 every 8 connection(s)
I20260929 20:30:19.253201 7206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
I20260929 20:30:19.256810 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 7206
I20260929 20:30:19.257038 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance
I20260929 20:30:19.257514 29650 external_mini_cluster.cc:1428] Setting key 99a541df47e558bdc781e37a49edfb12
I20260929 20:30:19.260681 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.131:0
--local_ip_for_outbound_sockets=127.28.244.131
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:33039
--builtin_ntp_servers=127.28.244.148:41169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 20:30:19.263234 7204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:19.256158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:56309 (local address 127.28.244.190:33039)
0929 20:30:19.256386 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:19.256391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:19.258371 (+ 1980us) server_negotiation.cc:407] Connection header received
0929 20:30:19.260412 (+ 2041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:19.260428 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:19.260503 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:19.260624 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:19.262265 (+ 1641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:30:19.262274 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:30:19.262282 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:30:19.262309 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:30:19.262355 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:30:19.262373 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:30:19.262541 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 20:30:19.262798 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 20:30:19.262890 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 20:30:19.264525 7335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:19.256831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:33039 (local address 127.28.244.130:56309)
0929 20:30:19.258192 (+ 1361us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:19.258243 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:19.260078 (+ 1835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:19.260914 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:19.260927 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:19.261065 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:19.261910 (+ 845us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:30:19.261913 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:19.261957 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:30:19.262060 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:30:19.262550 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:30:19.262562 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:30:19.262571 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 20:30:19.262870 (+ 299us) client_negotiation.cc:240] Negotiation successful
0929 20:30:19.263207 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":430,"threads_started":1}
I20260929 20:30:19.266181 7333 heartbeater.cc:347] Connected to a master server at 127.28.244.190:33039
I20260929 20:30:19.266554 7333 heartbeater.cc:464] Registering TS with master...
I20260929 20:30:19.267308 7333 heartbeater.cc:511] Master 127.28.244.190:33039 requested a full tablet report, sending...
I20260929 20:30:19.269052 7016 ts_manager.cc:194] Registered new tserver with Master: 7bb1a641a2164be2b79356b1b3d4ca84 (127.28.244.130:42687)
W20260929 20:30:19.606808 7336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:30:19.607141 7336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:30:19.607335 7336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:30:19.615751 7336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:30:19.615973 7336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:30:19.616156 7336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.131
I20260929 20:30:19.626804 7336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:41169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:33039
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.131
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:30:19.628319 7336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:30:19.630345 7336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:30:19.645563 7342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:30:19.645630 7341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:30:19.646070 7344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:30:19.646540 7336 server_base.cc:1034] running on GCE node
I20260929 20:30:19.647184 7336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:30:19.648326 7336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:30:19.649573 7336 hybrid_clock.cc:648] HybridClock initialized: now 1790713819649518 us; error 47 us; skew 500 ppm
I20260929 20:30:19.649935 7336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:30:19.652604 7336 webserver.cc:460] Webserver started at http://127.28.244.131:39769/ using document root <none> and password file <none>
I20260929 20:30:19.653403 7336 fs_manager.cc:359] Metadata directory not provided
I20260929 20:30:19.653551 7336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:30:19.653905 7336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:30:19.657226 7336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/data/instance:
uuid: "dc7534c8acce4a18a075028bb417e461"
format_stamp: "Formatted at 2026-09-29 20:30:19 on dist-test-slave-p29l"
server_key: "7cffb53c61918b44c3d5b00d1b7d59a0"
server_key_iv: "c4c4f5fd3093550387754d3911fb6156"
server_key_version: "encryptionkey@0"
I20260929 20:30:19.658190 7336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance:
uuid: "dc7534c8acce4a18a075028bb417e461"
format_stamp: "Formatted at 2026-09-29 20:30:19 on dist-test-slave-p29l"
server_key: "7cffb53c61918b44c3d5b00d1b7d59a0"
server_key_iv: "c4c4f5fd3093550387754d3911fb6156"
server_key_version: "encryptionkey@0"
I20260929 20:30:19.664332 7336 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260929 20:30:19.668982 7350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:30:19.670569 7336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:30:19.670807 7336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/wal
uuid: "dc7534c8acce4a18a075028bb417e461"
format_stamp: "Formatted at 2026-09-29 20:30:19 on dist-test-slave-p29l"
server_key: "7cffb53c61918b44c3d5b00d1b7d59a0"
server_key_iv: "c4c4f5fd3093550387754d3911fb6156"
server_key_version: "encryptionkey@0"
I20260929 20:30:19.671087 7336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:30:19.677451 7336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:30:19.678380 7336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:30:19.678673 7336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:30:19.679695 7336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:30:19.681281 7336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:30:19.681425 7336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:30:19.681563 7336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:30:19.681656 7336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:30:19.718164 7336 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.131:40677
I20260929 20:30:19.718204 7462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.131:40677 every 8 connection(s)
I20260929 20:30:19.719617 7336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
I20260929 20:30:19.726600 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 7336
I20260929 20:30:19.726818 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance
I20260929 20:30:19.727435 29650 external_mini_cluster.cc:1428] Setting key 56d59f164bbba16ee9ff9a273157738a
I20260929 20:30:19.728879 7204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:19.722620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:34653 (local address 127.28.244.190:33039)
0929 20:30:19.722809 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:19.722814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:19.724263 (+ 1449us) server_negotiation.cc:407] Connection header received
0929 20:30:19.726177 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:19.726183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:19.726263 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:19.726378 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:19.727972 (+ 1594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:30:19.727982 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:30:19.727988 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:30:19.728012 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:30:19.728054 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:30:19.728070 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:30:19.728207 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 20:30:19.728505 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 20:30:19.728580 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:30:19.729902 7465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:19.723043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:33039 (local address 127.28.244.131:34653)
0929 20:30:19.724091 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:19.724133 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:19.725892 (+ 1759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:19.726621 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:19.726630 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:19.726728 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:19.727570 (+ 842us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:30:19.727573 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:19.727628 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:30:19.727713 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:30:19.728208 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:30:19.728221 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:30:19.728230 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 20:30:19.728536 (+ 306us) client_negotiation.cc:240] Negotiation successful
0929 20:30:19.728805 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":284,"threads_started":1}
I20260929 20:30:19.731418 7463 heartbeater.cc:347] Connected to a master server at 127.28.244.190:33039
I20260929 20:30:19.731691 7463 heartbeater.cc:464] Registering TS with master...
I20260929 20:30:19.732372 7463 heartbeater.cc:511] Master 127.28.244.190:33039 requested a full tablet report, sending...
I20260929 20:30:19.733899 7016 ts_manager.cc:194] Registered new tserver with Master: dc7534c8acce4a18a075028bb417e461 (127.28.244.131:40677)
I20260929 20:30:19.744213 29650 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 20:30:19.750998 7204 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:30:19.749431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41978 (local address 127.28.244.190:33039)
0929 20:30:19.749632 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:19.749637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:19.750099 (+ 462us) server_negotiation.cc:407] Connection header received
0929 20:30:19.750248 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:19.750253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:19.750318 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:19.750417 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:19.750776 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41978: BlockingRecv error: recv got EOF from 127.0.0.1:41978 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 20:30:19.751559 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 7075
I20260929 20:30:19.755764 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:19.753124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35618 (local address 127.28.244.129:38917)
0929 20:30:19.753642 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:19.753648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:19.753676 (+ 28us) server_negotiation.cc:407] Connection header received
0929 20:30:19.753737 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:19.753744 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:19.753962 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:19.754112 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:19.755104 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:30:19.755111 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:30:19.755123 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:30:19.755143 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:30:19.755188 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:30:19.755203 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:30:19.755311 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 20:30:19.755414 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 20:30:19.755497 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":259,"threads_started":1}
I20260929 20:30:19.757117 7197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:30:19.839572 7202 heartbeater.cc:503] Master 127.28.244.190:33039 was elected leader, sending a full tablet report...
I20260929 20:30:19.841405 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 7075
I20260929 20:30:19.860066 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 7206
I20260929 20:30:19.864259 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:19.861088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42666 (local address 127.28.244.130:42687)
0929 20:30:19.861672 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:19.861680 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:19.861709 (+ 29us) server_negotiation.cc:407] Connection header received
0929 20:30:19.861774 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:19.861780 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:19.862017 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:19.862151 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:19.863185 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:30:19.863192 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:30:19.863205 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:30:19.863225 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:30:19.863269 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:30:19.863305 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:30:19.863465 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 20:30:19.863730 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 20:30:19.863889 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":283,"threads_started":1}
I20260929 20:30:19.866298 7328 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:30:19.952979 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 7206
I20260929 20:30:19.970000 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 7336
I20260929 20:30:19.974342 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:19.971147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58336 (local address 127.28.244.131:40677)
0929 20:30:19.971908 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:19.971917 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:19.971973 (+ 56us) server_negotiation.cc:407] Connection header received
0929 20:30:19.972084 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:19.972091 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:19.972343 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:19.972499 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:19.973567 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:30:19.973574 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:30:19.973586 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:30:19.973598 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:30:19.973641 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:30:19.973655 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:30:19.973790 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 20:30:19.973870 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 20:30:19.973958 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":394,"threads_started":1}
I20260929 20:30:19.976024 7458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:30:20.061086 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 7336
I20260929 20:30:20.078379 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 6985
I20260929 20:30:20.079758 7046 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:30:20.187080 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 6985
2026-09-29T20:30:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2522 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:30:20 dist-test-slave-p29l krb5kdc[7481](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:30:20 dist-test-slave-p29l krb5kdc[7481](info): set up 2 sockets
Sep 29 20:30:20 dist-test-slave-p29l krb5kdc[7481](info): commencing operation
krb5kdc: starting...
W20260929 20:30:21.344413 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:30:21 dist-test-slave-p29l krb5kdc[7481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:30:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:30:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.244.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
I20260929 20:30:21.597530 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:35649
--webserver_interface=127.28.244.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:45619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:35649
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:30:21.931367 7494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:30:21.931671 7494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:30:21.931767 7494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:30:21.939371 7494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:30:21.939486 7494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:30:21.939532 7494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:30:21.939591 7494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:30:21.939628 7494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:30:21.949827 7494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:35649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:35649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:30:21.951151 7494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:30:21.952435 7494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:30:21.964617 7499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:30:21.964536 7502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:30:21.964564 7500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:30:21.965399 7494 server_base.cc:1034] running on GCE node
I20260929 20:30:21.966267 7494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:30:21.967707 7494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:30:21.968923 7494 hybrid_clock.cc:648] HybridClock initialized: now 1790713821968854 us; error 55 us; skew 500 ppm
I20260929 20:30:21.969291 7494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:30:21 dist-test-slave-p29l krb5kdc[7481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713821, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:30:21.972491 7494 init.cc:377] Logged in from keytab as kudu/127.28.244.190@KRBTEST.COM (short username kudu)
I20260929 20:30:21.973863 7494 webserver.cc:460] Webserver started at http://127.28.244.190:45091/ using document root <none> and password file <none>
I20260929 20:30:21.974475 7494 fs_manager.cc:359] Metadata directory not provided
I20260929 20:30:21.974570 7494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:30:21.974822 7494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:30:21.977695 7494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "f1476027fabd48c7b1a4a97a92a0fbcf"
format_stamp: "Formatted at 2026-09-29 20:30:21 on dist-test-slave-p29l"
server_key: "cd7bff2e2fc3da0bc91db869edc267d4"
server_key_iv: "643e4d86cc0411bba1b7b9565d5fa231"
server_key_version: "encryptionkey@0"
I20260929 20:30:21.978375 7494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "f1476027fabd48c7b1a4a97a92a0fbcf"
format_stamp: "Formatted at 2026-09-29 20:30:21 on dist-test-slave-p29l"
server_key: "cd7bff2e2fc3da0bc91db869edc267d4"
server_key_iv: "643e4d86cc0411bba1b7b9565d5fa231"
server_key_version: "encryptionkey@0"
I20260929 20:30:21.983521 7494 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:30:21.987006 7509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:30:21.988633 7494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:30:21.988828 7494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "f1476027fabd48c7b1a4a97a92a0fbcf"
format_stamp: "Formatted at 2026-09-29 20:30:21 on dist-test-slave-p29l"
server_key: "cd7bff2e2fc3da0bc91db869edc267d4"
server_key_iv: "643e4d86cc0411bba1b7b9565d5fa231"
server_key_version: "encryptionkey@0"
I20260929 20:30:21.989039 7494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:30:22.011852 7494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:30:22.016139 7494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:30:22.016427 7494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:30:22.035429 7494 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:35649
I20260929 20:30:22.035466 7560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:35649 every 8 connection(s)
I20260929 20:30:22.036834 7494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:30:22.040380 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 7494
I20260929 20:30:22.040607 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:30:22.040885 7561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:30:22.041280 29650 external_mini_cluster.cc:1428] Setting key e751d50405e9f021e3379243c7e84dfe
I20260929 20:30:22.055934 7561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1476027fabd48c7b1a4a97a92a0fbcf: Bootstrap starting.
Sep 29 20:30:22 dist-test-slave-p29l krb5kdc[7481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:30:22.060218 7561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1476027fabd48c7b1a4a97a92a0fbcf: Neither blocks nor log segments found. Creating new log.
I20260929 20:30:22.061626 7561 log.cc:824] T 00000000000000000000000000000000 P f1476027fabd48c7b1a4a97a92a0fbcf: Log is configured to *not* fsync() on all Append() calls
I20260929 20:30:22.065655 7561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1476027fabd48c7b1a4a97a92a0fbcf: No bootstrap required, opened a new log
I20260929 20:30:22.066447 7564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:22.043287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50246 (local address 127.28.244.190:35649)
0929 20:30:22.044212 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:22.044228 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:22.044289 (+ 61us) server_negotiation.cc:407] Connection header received
0929 20:30:22.046013 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:22.046051 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:22.046607 (+ 556us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:22.047082 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:22.048416 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:22.050422 (+ 2006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:22.051713 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:22.052879 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:22.053084 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:22.057064 (+ 3980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:30:22.057076 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:30:22.057097 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:30:22.057152 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:30:22.061217 (+ 4065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:30:22.062020 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:30:22.062030 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:30:22.062039 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:30:22.062303 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:30:22.062950 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:30:22.062954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:30:22.062957 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:30:22.063897 (+ 940us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:30:22.064213 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0929 20:30:22.064571 (+ 358us) server_negotiation.cc:299] Negotiation successful
0929 20:30:22.065043 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":351,"threads_started":1}
I20260929 20:30:22.074563 7561 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1476027fabd48c7b1a4a97a92a0fbcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1476027fabd48c7b1a4a97a92a0fbcf" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 35649 } }
I20260929 20:30:22.074941 7561 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1476027fabd48c7b1a4a97a92a0fbcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:30:22.075016 7561 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1476027fabd48c7b1a4a97a92a0fbcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1476027fabd48c7b1a4a97a92a0fbcf, State: Initialized, Role: FOLLOWER
I20260929 20:30:22.075575 7561 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1476027fabd48c7b1a4a97a92a0fbcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1476027fabd48c7b1a4a97a92a0fbcf" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 35649 } }
I20260929 20:30:22.075773 7561 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1476027fabd48c7b1a4a97a92a0fbcf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:30:22.075958 7561 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1476027fabd48c7b1a4a97a92a0fbcf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:30:22.076090 7561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1476027fabd48c7b1a4a97a92a0fbcf [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:30:22.078665 7561 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1476027fabd48c7b1a4a97a92a0fbcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1476027fabd48c7b1a4a97a92a0fbcf" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 35649 } }
I20260929 20:30:22.079147 7561 leader_election.cc:302] T 00000000000000000000000000000000 P f1476027fabd48c7b1a4a97a92a0fbcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1476027fabd48c7b1a4a97a92a0fbcf; no voters:
I20260929 20:30:22.079664 7561 leader_election.cc:288] T 00000000000000000000000000000000 P f1476027fabd48c7b1a4a97a92a0fbcf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:30:22.079831 7566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1476027fabd48c7b1a4a97a92a0fbcf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:30:22.080709 7566 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1476027fabd48c7b1a4a97a92a0fbcf [term 1 LEADER]: Becoming Leader. State: Replica: f1476027fabd48c7b1a4a97a92a0fbcf, State: Running, Role: LEADER
I20260929 20:30:22.081300 7566 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1476027fabd48c7b1a4a97a92a0fbcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1476027fabd48c7b1a4a97a92a0fbcf" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 35649 } }
I20260929 20:30:22.082150 7561 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1476027fabd48c7b1a4a97a92a0fbcf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:30:22.084257 7568 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1476027fabd48c7b1a4a97a92a0fbcf [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1476027fabd48c7b1a4a97a92a0fbcf. Latest consensus state: current_term: 1 leader_uuid: "f1476027fabd48c7b1a4a97a92a0fbcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1476027fabd48c7b1a4a97a92a0fbcf" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 35649 } } }
I20260929 20:30:22.084270 7567 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1476027fabd48c7b1a4a97a92a0fbcf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1476027fabd48c7b1a4a97a92a0fbcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1476027fabd48c7b1a4a97a92a0fbcf" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 35649 } } }
I20260929 20:30:22.084674 7568 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1476027fabd48c7b1a4a97a92a0fbcf [sys.catalog]: This master's current role is: LEADER
I20260929 20:30:22.084681 7567 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1476027fabd48c7b1a4a97a92a0fbcf [sys.catalog]: This master's current role is: LEADER
I20260929 20:30:22.085565 7571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:30:22.091323 7571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:30:22.100558 7571 catalog_manager.cc:1347] Generated new cluster ID: 1f0bfd7e28fc456f91db5245e071df95
I20260929 20:30:22.100698 7571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:30:22.115353 7571 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:30:22.116789 7571 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:30:22.142967 7571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1476027fabd48c7b1a4a97a92a0fbcf: Generated new TSK 0
I20260929 20:30:22.144012 7571 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.244.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab.
Entry for principal kudu/127.28.244.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab.
I20260929 20:30:22.208048 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.129:0
--local_ip_for_outbound_sockets=127.28.244.129
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:35649
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:45619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:30:22.559733 7587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:30:22.560148 7587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:30:22.560320 7587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:30:22.568336 7587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:30:22.568511 7587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:30:22.568687 7587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.129
I20260929 20:30:22.579388 7587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:35649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.129
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:30:22.580880 7587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:30:22.582316 7587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:30:22.596783 7595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:30:22.596853 7592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:30:22.596881 7593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:30:22.597304 7587 server_base.cc:1034] running on GCE node
I20260929 20:30:22.598117 7587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:30:22.599225 7587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:30:22.600455 7587 hybrid_clock.cc:648] HybridClock initialized: now 1790713822600408 us; error 29 us; skew 500 ppm
I20260929 20:30:22.600800 7587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:30:22 dist-test-slave-p29l krb5kdc[7481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713822, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:30:22.611071 7587 init.cc:377] Logged in from keytab as kudu/127.28.244.129@KRBTEST.COM (short username kudu)
I20260929 20:30:22.612696 7587 webserver.cc:460] Webserver started at http://127.28.244.129:38903/ using document root <none> and password file <none>
I20260929 20:30:22.613404 7587 fs_manager.cc:359] Metadata directory not provided
I20260929 20:30:22.613503 7587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:30:22.613757 7587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:30:22.616816 7587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/data/instance:
uuid: "dbda93b240994b278bf354743af462b2"
format_stamp: "Formatted at 2026-09-29 20:30:22 on dist-test-slave-p29l"
server_key: "70f976f33f2a0fecc1899c489b450e17"
server_key_iv: "80c1c37d8e82886613aeceb40e5b601d"
server_key_version: "encryptionkey@0"
I20260929 20:30:22.617599 7587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance:
uuid: "dbda93b240994b278bf354743af462b2"
format_stamp: "Formatted at 2026-09-29 20:30:22 on dist-test-slave-p29l"
server_key: "70f976f33f2a0fecc1899c489b450e17"
server_key_iv: "80c1c37d8e82886613aeceb40e5b601d"
server_key_version: "encryptionkey@0"
I20260929 20:30:22.622874 7587 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:30:22.626511 7602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:30:22.628082 7587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:30:22.628288 7587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/wal
uuid: "dbda93b240994b278bf354743af462b2"
format_stamp: "Formatted at 2026-09-29 20:30:22 on dist-test-slave-p29l"
server_key: "70f976f33f2a0fecc1899c489b450e17"
server_key_iv: "80c1c37d8e82886613aeceb40e5b601d"
server_key_version: "encryptionkey@0"
I20260929 20:30:22.628523 7587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:30:22.666043 7587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:30:22.670079 7587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:30:22.670392 7587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:30:22.671381 7587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:30:22.672895 7587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:30:22.673075 7587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 20:30:22.673209 7587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:30:22.673297 7587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:30:22.707460 7587 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.129:43905
I20260929 20:30:22.707496 7714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.129:43905 every 8 connection(s)
I20260929 20:30:22.708781 7587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
I20260929 20:30:22.715268 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 7587
I20260929 20:30:22.715499 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance
I20260929 20:30:22.716312 29650 external_mini_cluster.cc:1428] Setting key 5ad35cd9150025c6eba3b662b16f243d
Sep 29 20:30:22 dist-test-slave-p29l krb5kdc[7481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713822, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:30:22.730016 7717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:22.711242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:39743 (local address 127.28.244.190:35649)
0929 20:30:22.711690 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:22.711694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:22.712721 (+ 1027us) server_negotiation.cc:407] Connection header received
0929 20:30:22.715338 (+ 2617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:22.715345 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:22.715497 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:22.715685 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:22.717614 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:22.718334 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:22.719901 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:22.720311 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:22.720466 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:22.724246 (+ 3780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:30:22.724251 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:30:22.724255 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:30:22.724288 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:30:22.726878 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:30:22.727655 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:30:22.727660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:30:22.727663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:30:22.727777 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:30:22.728266 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:30:22.728271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:30:22.728274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:30:22.728541 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:30:22.728705 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 20:30:22.729422 (+ 717us) server_negotiation.cc:299] Negotiation successful
0929 20:30:22.729658 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":218,"threads_started":1}
I20260929 20:30:22.730996 7718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:22.711780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35649 (local address 127.28.244.129:39743)
0929 20:30:22.712532 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:22.712575 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:22.714981 (+ 2406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:22.715983 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:22.715995 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:22.716743 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:22.717429 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:22.717447 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:22.718490 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:22.718495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:22.719726 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:22.719738 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:22.720553 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:22.720560 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:22.720815 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:22.721519 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:22.721543 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:22.724001 (+ 2458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:30:22.727051 (+ 3050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:30:22.727060 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:30:22.727087 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:30:22.727509 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:30:22.727932 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:30:22.727938 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:30:22.727941 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:30:22.728141 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:30:22.728676 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:30:22.728685 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:30:22.729032 (+ 347us) client_negotiation.cc:769] Sending connection context
0929 20:30:22.729304 (+ 272us) client_negotiation.cc:240] Negotiation successful
0929 20:30:22.729833 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":266,"threads_started":1}
I20260929 20:30:22.732801 7715 heartbeater.cc:347] Connected to a master server at 127.28.244.190:35649
I20260929 20:30:22.733152 7715 heartbeater.cc:464] Registering TS with master...
I20260929 20:30:22.734086 7715 heartbeater.cc:511] Master 127.28.244.190:35649 requested a full tablet report, sending...
I20260929 20:30:22.736760 7526 ts_manager.cc:194] Registered new tserver with Master: dbda93b240994b278bf354743af462b2 (127.28.244.129:43905)
I20260929 20:30:22.738996 7526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.129@KRBTEST.COM'} at 127.28.244.129:39743
WARNING: no policy specified for kudu/127.28.244.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab.
Entry for principal kudu/127.28.244.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab.
I20260929 20:30:22.771150 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.130:0
--local_ip_for_outbound_sockets=127.28.244.130
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:35649
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:45619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:30:23.108801 7721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:30:23.109189 7721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:30:23.109395 7721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:30:23.117667 7721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:30:23.117859 7721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:30:23.118072 7721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.130
I20260929 20:30:23.129709 7721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:35649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.130
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:30:23.131319 7721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:30:23.133061 7721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:30:23.147955 7727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:30:23.148000 7726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:30:23.148048 7729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:30:23.148803 7721 server_base.cc:1034] running on GCE node
I20260929 20:30:23.149412 7721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:30:23.150346 7721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:30:23.151541 7721 hybrid_clock.cc:648] HybridClock initialized: now 1790713823151475 us; error 60 us; skew 500 ppm
I20260929 20:30:23.151918 7721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:30:23 dist-test-slave-p29l krb5kdc[7481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713823, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:30:23.155032 7721 init.cc:377] Logged in from keytab as kudu/127.28.244.130@KRBTEST.COM (short username kudu)
I20260929 20:30:23.156564 7721 webserver.cc:460] Webserver started at http://127.28.244.130:38689/ using document root <none> and password file <none>
I20260929 20:30:23.157260 7721 fs_manager.cc:359] Metadata directory not provided
I20260929 20:30:23.157416 7721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:30:23.157724 7721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:30:23.160790 7721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/data/instance:
uuid: "c7947c42ec424c058bfd54779eb4b8c4"
format_stamp: "Formatted at 2026-09-29 20:30:23 on dist-test-slave-p29l"
server_key: "4ab9dc0c9d70606ec0dfee90c39b0ab6"
server_key_iv: "5afea98a61ea0777ea1a7137e648d07a"
server_key_version: "encryptionkey@0"
I20260929 20:30:23.161586 7721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance:
uuid: "c7947c42ec424c058bfd54779eb4b8c4"
format_stamp: "Formatted at 2026-09-29 20:30:23 on dist-test-slave-p29l"
server_key: "4ab9dc0c9d70606ec0dfee90c39b0ab6"
server_key_iv: "5afea98a61ea0777ea1a7137e648d07a"
server_key_version: "encryptionkey@0"
I20260929 20:30:23.166785 7721 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 20:30:23.170586 7736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:30:23.172055 7721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:30:23.172231 7721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/wal
uuid: "c7947c42ec424c058bfd54779eb4b8c4"
format_stamp: "Formatted at 2026-09-29 20:30:23 on dist-test-slave-p29l"
server_key: "4ab9dc0c9d70606ec0dfee90c39b0ab6"
server_key_iv: "5afea98a61ea0777ea1a7137e648d07a"
server_key_version: "encryptionkey@0"
I20260929 20:30:23.172415 7721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:30:23.197044 7721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:30:23.201395 7721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:30:23.201710 7721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:30:23.202742 7721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:30:23.204313 7721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:30:23.204388 7721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:30:23.204552 7721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:30:23.204617 7721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:30:23.238960 7721 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.130:43557
I20260929 20:30:23.239037 7848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.130:43557 every 8 connection(s)
I20260929 20:30:23.240353 7721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
I20260929 20:30:23.244837 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 7721
I20260929 20:30:23.245043 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance
I20260929 20:30:23.245668 29650 external_mini_cluster.cc:1428] Setting key 6093f626b75a4a44eaf5c4bae9b1209c
Sep 29 20:30:23 dist-test-slave-p29l krb5kdc[7481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713823, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:30:23.271004 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:23.242915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:36963 (local address 127.28.244.190:35649)
0929 20:30:23.243320 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:23.243324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:23.244525 (+ 1201us) server_negotiation.cc:407] Connection header received
0929 20:30:23.246858 (+ 2333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:23.246863 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:23.246939 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:23.247068 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:23.249026 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:23.249781 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:23.252435 (+ 2654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:23.252884 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:23.252993 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:23.262766 (+ 9773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:30:23.262777 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:30:23.262785 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:30:23.262840 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:30:23.266774 (+ 3934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:30:23.268046 (+ 1272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:30:23.268055 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:30:23.268061 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:30:23.268266 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:30:23.268786 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:30:23.268794 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:30:23.268801 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:30:23.269217 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:30:23.269381 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 20:30:23.270273 (+ 892us) server_negotiation.cc:299] Negotiation successful
0929 20:30:23.270530 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":199,"threads_started":1}
WARNING: no policy specified for kudu/127.28.244.131@KRBTEST.COM; defaulting to no policy
I20260929 20:30:23.272076 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:23.243449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35649 (local address 127.28.244.130:36963)
0929 20:30:23.244287 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:23.244358 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:23.246601 (+ 2243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:23.247270 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:23.247279 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:23.247935 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:23.248820 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:23.248850 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:23.249953 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:23.249961 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:23.252238 (+ 2277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:23.252258 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:23.253144 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:23.253153 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:23.253515 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:23.259673 (+ 6158us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:23.259706 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:23.262397 (+ 2691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:30:23.267008 (+ 4611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:30:23.267018 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:30:23.267035 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:30:23.267770 (+ 735us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:30:23.268420 (+ 650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:30:23.268425 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:30:23.268427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:30:23.268640 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:30:23.269380 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:30:23.269390 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:30:23.269809 (+ 419us) client_negotiation.cc:769] Sending connection context
0929 20:30:23.270099 (+ 290us) client_negotiation.cc:240] Negotiation successful
0929 20:30:23.270569 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":386,"threads_started":1}
I20260929 20:30:23.273869 7849 heartbeater.cc:347] Connected to a master server at 127.28.244.190:35649
I20260929 20:30:23.274300 7849 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.131@KRBTEST.COM" created.
I20260929 20:30:23.275343 7849 heartbeater.cc:511] Master 127.28.244.190:35649 requested a full tablet report, sending...
I20260929 20:30:23.277428 7526 ts_manager.cc:194] Registered new tserver with Master: c7947c42ec424c058bfd54779eb4b8c4 (127.28.244.130:43557)
I20260929 20:30:23.278784 7526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.130@KRBTEST.COM'} at 127.28.244.130:36963
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab.
Entry for principal kudu/127.28.244.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab.
I20260929 20:30:23.307579 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.131:0
--local_ip_for_outbound_sockets=127.28.244.131
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:35649
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:45619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:30:23.660358 7855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:30:23.660737 7855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:30:23.660913 7855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:30:23.668177 7855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:30:23.668341 7855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:30:23.668533 7855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.131
I20260929 20:30:23.679093 7855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:35649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.131
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:30:23.680593 7855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:30:23.682191 7855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:30:23.696686 7861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:30:23.696791 7863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:30:23.696735 7860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:30:23.697719 7855 server_base.cc:1034] running on GCE node
I20260929 20:30:23.698443 7855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:30:23.699494 7855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:30:23.700758 7855 hybrid_clock.cc:648] HybridClock initialized: now 1790713823700710 us; error 34 us; skew 500 ppm
I20260929 20:30:23.701135 7855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:30:23 dist-test-slave-p29l krb5kdc[7481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713823, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:30:23.704329 7855 init.cc:377] Logged in from keytab as kudu/127.28.244.131@KRBTEST.COM (short username kudu)
I20260929 20:30:23.705885 7855 webserver.cc:460] Webserver started at http://127.28.244.131:41021/ using document root <none> and password file <none>
I20260929 20:30:23.706599 7855 fs_manager.cc:359] Metadata directory not provided
I20260929 20:30:23.706743 7855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:30:23.707113 7855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:30:23.710098 7855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/data/instance:
uuid: "b580081cd50c4c80a6ebad431cb62043"
format_stamp: "Formatted at 2026-09-29 20:30:23 on dist-test-slave-p29l"
server_key: "0736a65405d3c9b5e13064baed5b63b9"
server_key_iv: "4347c8fa695cbe20391645dcd75812a5"
server_key_version: "encryptionkey@0"
I20260929 20:30:23.710865 7855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance:
uuid: "b580081cd50c4c80a6ebad431cb62043"
format_stamp: "Formatted at 2026-09-29 20:30:23 on dist-test-slave-p29l"
server_key: "0736a65405d3c9b5e13064baed5b63b9"
server_key_iv: "4347c8fa695cbe20391645dcd75812a5"
server_key_version: "encryptionkey@0"
I20260929 20:30:23.716012 7855 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 20:30:23.720715 7870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:30:23.722275 7855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:30:23.722434 7855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/wal
uuid: "b580081cd50c4c80a6ebad431cb62043"
format_stamp: "Formatted at 2026-09-29 20:30:23 on dist-test-slave-p29l"
server_key: "0736a65405d3c9b5e13064baed5b63b9"
server_key_iv: "4347c8fa695cbe20391645dcd75812a5"
server_key_version: "encryptionkey@0"
I20260929 20:30:23.722693 7855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:30:23.734061 7855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:30:23.738523 7855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:30:23.738835 7855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:30:23.739864 7855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:30:23.741312 7855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:30:23.741386 7855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:30:23.741524 7855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:30:23.741629 7855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:30:23.742215 7715 heartbeater.cc:503] Master 127.28.244.190:35649 was elected leader, sending a full tablet report...
I20260929 20:30:23.774811 7855 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.131:34645
I20260929 20:30:23.774858 7982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.131:34645 every 8 connection(s)
I20260929 20:30:23.776351 7855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
I20260929 20:30:23.781556 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 7855
I20260929 20:30:23.781775 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance
I20260929 20:30:23.782212 29650 external_mini_cluster.cc:1428] Setting key 2d1c8c7e2ff9e39fcb1a4e90c7714993
Sep 29 20:30:23 dist-test-slave-p29l krb5kdc[7481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713823, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:30:23.798069 7985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:23.779024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:43619 (local address 127.28.244.190:35649)
0929 20:30:23.779478 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:23.779483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:23.780466 (+ 983us) server_negotiation.cc:407] Connection header received
0929 20:30:23.782505 (+ 2039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:23.782511 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:23.782637 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:23.782791 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:23.784790 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:23.785720 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:23.787371 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:23.787869 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:23.788010 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:23.791588 (+ 3578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:30:23.791594 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:30:23.791598 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:30:23.791633 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:30:23.794458 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:30:23.795254 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:30:23.795259 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:30:23.795263 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:30:23.795397 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:30:23.795871 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:30:23.795876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:30:23.795879 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:30:23.796163 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:30:23.796347 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 20:30:23.797353 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0929 20:30:23.797679 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":217,"threads_started":1}
I20260929 20:30:23.799304 7986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:23.779566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35649 (local address 127.28.244.131:43619)
0929 20:30:23.780300 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:23.780338 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:23.782173 (+ 1835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:23.783118 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:23.783132 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:23.783904 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:30:23.784619 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:23.784641 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:23.785881 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:23.785885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:23.787222 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:23.787235 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:23.788108 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:23.788115 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:23.788369 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:23.789039 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:30:23.789061 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:30:23.791337 (+ 2276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:30:23.794626 (+ 3289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:30:23.794637 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:30:23.794657 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:30:23.795123 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:30:23.795516 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:30:23.795521 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:30:23.795525 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:30:23.795726 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:30:23.796357 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:30:23.796364 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:30:23.796905 (+ 541us) client_negotiation.cc:769] Sending connection context
0929 20:30:23.797272 (+ 367us) client_negotiation.cc:240] Negotiation successful
0929 20:30:23.798043 (+ 771us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":308,"threads_started":1}
I20260929 20:30:23.800877 7983 heartbeater.cc:347] Connected to a master server at 127.28.244.190:35649
I20260929 20:30:23.801234 7983 heartbeater.cc:464] Registering TS with master...
I20260929 20:30:23.802038 7983 heartbeater.cc:511] Master 127.28.244.190:35649 requested a full tablet report, sending...
I20260929 20:30:23.803654 7526 ts_manager.cc:194] Registered new tserver with Master: b580081cd50c4c80a6ebad431cb62043 (127.28.244.131:34645)
I20260929 20:30:23.804669 7526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.131@KRBTEST.COM'} at 127.28.244.131:43619
I20260929 20:30:23.810843 29650 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:30:23.827680 7985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:23.815461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50258 (local address 127.28.244.190:35649)
0929 20:30:23.815653 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:23.815658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:23.816258 (+ 600us) server_negotiation.cc:407] Connection header received
0929 20:30:23.816421 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:23.816425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:23.816508 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:23.816615 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:23.817984 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:23.818738 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:23.820065 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:23.820434 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:23.820534 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:23.822768 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:30:23.822773 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:30:23.822777 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:30:23.822803 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:30:23.825260 (+ 2457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:30:23.825876 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:30:23.825880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:30:23.825882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:30:23.825989 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:30:23.826315 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:30:23.826318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:30:23.826320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:30:23.826704 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:30:23.826893 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 20:30:23.827179 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 20:30:23.827371 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:30:23.831866 7526 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50258:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:30:23.839089 7526 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:30:23.864121 7996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:23.852130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:43557 (local address 127.0.0.1:36408)
0929 20:30:23.853044 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:23.853094 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:23.853315 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:23.855258 (+ 1943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:23.855270 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:23.855313 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:30:23.855865 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:23.855884 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:23.858899 (+ 3015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:23.858909 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:23.860925 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:23.860941 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:23.862925 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:23.862934 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:23.863207 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:23.863283 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 20:30:23.863384 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 20:30:23.863605 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":363,"threads_started":1}
I20260929 20:30:23.864790 7998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:23.852275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36408 (local address 127.28.244.130:43557)
0929 20:30:23.854434 (+ 2159us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:23.854446 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:23.854482 (+ 36us) server_negotiation.cc:407] Connection header received
0929 20:30:23.854601 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:23.854612 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:23.854944 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:23.856171 (+ 1227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:30:23.856486 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:23.858671 (+ 2185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:23.861106 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:23.862674 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:23.863958 (+ 1284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:23.864174 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 20:30:23.864251 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 20:30:23.864443 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1949,"thread_start_us":1755,"threads_started":1}
I20260929 20:30:23.868892 7784 tablet_service.cc:1467] Processing CreateTablet for tablet 6b21f25189604ca495d4b4bc0adda37f (DEFAULT_TABLE table=test-table [id=febe1082ea0b46b4bf87cf45fb47fa68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:30:23.870134 7997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:23.852204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35658 (local address 127.28.244.129:43905)
0929 20:30:23.859879 (+ 7675us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:23.859893 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:23.861820 (+ 1927us) server_negotiation.cc:407] Connection header received
0929 20:30:23.861935 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:23.861944 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:23.862256 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:23.862459 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:30:23.863445 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:23.865321 (+ 1876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:23.867708 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:23.869022 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:23.869239 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:23.869408 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 20:30:23.869613 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 20:30:23.869769 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7421,"thread_start_us":7208,"threads_started":1}
I20260929 20:30:23.870143 7999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:23.852700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:43905 (local address 127.0.0.1:35658)
0929 20:30:23.861538 (+ 8838us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:23.861560 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:23.861697 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:23.862710 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:23.862717 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:23.862781 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:30:23.863240 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:23.863252 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:23.865450 (+ 2198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:23.865458 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:23.867518 (+ 2060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:23.867533 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:23.869238 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:23.869244 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:23.869406 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:23.869446 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 20:30:23.869568 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 20:30:23.869705 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8700,"mutex_wait_us":796,"spinlock_wait_cycles":21888,"thread_start_us":7719,"threads_started":1}
I20260929 20:30:23.870425 7784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b21f25189604ca495d4b4bc0adda37f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:30:23.872077 8001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:23.855499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35214 (local address 127.28.244.131:34645)
0929 20:30:23.862156 (+ 6657us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:23.862168 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:23.862916 (+ 748us) server_negotiation.cc:407] Connection header received
0929 20:30:23.863021 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:23.863029 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:23.863275 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:23.863478 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:30:23.864323 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:23.865882 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:23.868613 (+ 2731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:23.870057 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:23.870560 (+ 503us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:23.870736 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 20:30:23.871559 (+ 823us) server_negotiation.cc:299] Negotiation successful
0929 20:30:23.871736 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6466,"thread_start_us":6323,"threads_started":1}
I20260929 20:30:23.872752 8000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:23.854896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:34645 (local address 127.0.0.1:35214)
0929 20:30:23.862618 (+ 7722us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:23.862635 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:23.862775 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:23.863617 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:23.863622 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:23.863647 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:30:23.864109 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:23.864122 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:23.866334 (+ 2212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:23.866341 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:23.868424 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:23.868440 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:23.870982 (+ 2542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:23.870988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:23.871169 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:23.871202 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 20:30:23.871286 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 20:30:23.872345 (+ 1059us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7590,"spinlock_wait_cycles":1792,"thread_start_us":7446,"threads_started":1}
I20260929 20:30:23.874300 7650 tablet_service.cc:1467] Processing CreateTablet for tablet 6b21f25189604ca495d4b4bc0adda37f (DEFAULT_TABLE table=test-table [id=febe1082ea0b46b4bf87cf45fb47fa68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:30:23.876008 7650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b21f25189604ca495d4b4bc0adda37f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:30:23.877116 7918 tablet_service.cc:1467] Processing CreateTablet for tablet 6b21f25189604ca495d4b4bc0adda37f (DEFAULT_TABLE table=test-table [id=febe1082ea0b46b4bf87cf45fb47fa68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:30:23.878599 7918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b21f25189604ca495d4b4bc0adda37f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:30:23.885250 8002 tablet_bootstrap.cc:492] T 6b21f25189604ca495d4b4bc0adda37f P c7947c42ec424c058bfd54779eb4b8c4: Bootstrap starting.
I20260929 20:30:23.889657 8002 tablet_bootstrap.cc:654] T 6b21f25189604ca495d4b4bc0adda37f P c7947c42ec424c058bfd54779eb4b8c4: Neither blocks nor log segments found. Creating new log.
I20260929 20:30:23.890161 8003 tablet_bootstrap.cc:492] T 6b21f25189604ca495d4b4bc0adda37f P dbda93b240994b278bf354743af462b2: Bootstrap starting.
I20260929 20:30:23.891983 8002 log.cc:824] T 6b21f25189604ca495d4b4bc0adda37f P c7947c42ec424c058bfd54779eb4b8c4: Log is configured to *not* fsync() on all Append() calls
I20260929 20:30:23.894161 8003 tablet_bootstrap.cc:654] T 6b21f25189604ca495d4b4bc0adda37f P dbda93b240994b278bf354743af462b2: Neither blocks nor log segments found. Creating new log.
I20260929 20:30:23.894486 8005 tablet_bootstrap.cc:492] T 6b21f25189604ca495d4b4bc0adda37f P b580081cd50c4c80a6ebad431cb62043: Bootstrap starting.
I20260929 20:30:23.895452 8003 log.cc:824] T 6b21f25189604ca495d4b4bc0adda37f P dbda93b240994b278bf354743af462b2: Log is configured to *not* fsync() on all Append() calls
I20260929 20:30:23.895613 8002 tablet_bootstrap.cc:492] T 6b21f25189604ca495d4b4bc0adda37f P c7947c42ec424c058bfd54779eb4b8c4: No bootstrap required, opened a new log
I20260929 20:30:23.895969 8002 ts_tablet_manager.cc:1397] T 6b21f25189604ca495d4b4bc0adda37f P c7947c42ec424c058bfd54779eb4b8c4: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260929 20:30:23.898537 8005 tablet_bootstrap.cc:654] T 6b21f25189604ca495d4b4bc0adda37f P b580081cd50c4c80a6ebad431cb62043: Neither blocks nor log segments found. Creating new log.
I20260929 20:30:23.899887 8005 log.cc:824] T 6b21f25189604ca495d4b4bc0adda37f P b580081cd50c4c80a6ebad431cb62043: Log is configured to *not* fsync() on all Append() calls
I20260929 20:30:23.901649 8003 tablet_bootstrap.cc:492] T 6b21f25189604ca495d4b4bc0adda37f P dbda93b240994b278bf354743af462b2: No bootstrap required, opened a new log
I20260929 20:30:23.901926 8003 ts_tablet_manager.cc:1397] T 6b21f25189604ca495d4b4bc0adda37f P dbda93b240994b278bf354743af462b2: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260929 20:30:23.902992 8005 tablet_bootstrap.cc:492] T 6b21f25189604ca495d4b4bc0adda37f P b580081cd50c4c80a6ebad431cb62043: No bootstrap required, opened a new log
I20260929 20:30:23.903271 8005 ts_tablet_manager.cc:1397] T 6b21f25189604ca495d4b4bc0adda37f P b580081cd50c4c80a6ebad431cb62043: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260929 20:30:23.906551 8002 raft_consensus.cc:348] T 6b21f25189604ca495d4b4bc0adda37f P c7947c42ec424c058bfd54779eb4b8c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbda93b240994b278bf354743af462b2" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 43905 } } peers { permanent_uuid: "c7947c42ec424c058bfd54779eb4b8c4" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 43557 } } peers { permanent_uuid: "b580081cd50c4c80a6ebad431cb62043" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 34645 } }
I20260929 20:30:23.907037 8002 raft_consensus.cc:374] T 6b21f25189604ca495d4b4bc0adda37f P c7947c42ec424c058bfd54779eb4b8c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:30:23.907140 8002 raft_consensus.cc:729] T 6b21f25189604ca495d4b4bc0adda37f P c7947c42ec424c058bfd54779eb4b8c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7947c42ec424c058bfd54779eb4b8c4, State: Initialized, Role: FOLLOWER
I20260929 20:30:23.907894 8002 consensus_queue.cc:260] T 6b21f25189604ca495d4b4bc0adda37f P c7947c42ec424c058bfd54779eb4b8c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbda93b240994b278bf354743af462b2" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 43905 } } peers { permanent_uuid: "c7947c42ec424c058bfd54779eb4b8c4" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 43557 } } peers { permanent_uuid: "b580081cd50c4c80a6ebad431cb62043" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 34645 } }
I20260929 20:30:23.909113 7849 heartbeater.cc:503] Master 127.28.244.190:35649 was elected leader, sending a full tablet report...
I20260929 20:30:23.909870 8002 ts_tablet_manager.cc:1428] T 6b21f25189604ca495d4b4bc0adda37f P c7947c42ec424c058bfd54779eb4b8c4: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260929 20:30:23.910454 8003 raft_consensus.cc:348] T 6b21f25189604ca495d4b4bc0adda37f P dbda93b240994b278bf354743af462b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbda93b240994b278bf354743af462b2" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 43905 } } peers { permanent_uuid: "c7947c42ec424c058bfd54779eb4b8c4" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 43557 } } peers { permanent_uuid: "b580081cd50c4c80a6ebad431cb62043" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 34645 } }
I20260929 20:30:23.910903 8003 raft_consensus.cc:374] T 6b21f25189604ca495d4b4bc0adda37f P dbda93b240994b278bf354743af462b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:30:23.911020 8003 raft_consensus.cc:729] T 6b21f25189604ca495d4b4bc0adda37f P dbda93b240994b278bf354743af462b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbda93b240994b278bf354743af462b2, State: Initialized, Role: FOLLOWER
I20260929 20:30:23.911674 8003 consensus_queue.cc:260] T 6b21f25189604ca495d4b4bc0adda37f P dbda93b240994b278bf354743af462b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbda93b240994b278bf354743af462b2" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 43905 } } peers { permanent_uuid: "c7947c42ec424c058bfd54779eb4b8c4" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 43557 } } peers { permanent_uuid: "b580081cd50c4c80a6ebad431cb62043" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 34645 } }
I20260929 20:30:23.914947 8005 raft_consensus.cc:348] T 6b21f25189604ca495d4b4bc0adda37f P b580081cd50c4c80a6ebad431cb62043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbda93b240994b278bf354743af462b2" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 43905 } } peers { permanent_uuid: "c7947c42ec424c058bfd54779eb4b8c4" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 43557 } } peers { permanent_uuid: "b580081cd50c4c80a6ebad431cb62043" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 34645 } }
I20260929 20:30:23.915454 8005 raft_consensus.cc:374] T 6b21f25189604ca495d4b4bc0adda37f P b580081cd50c4c80a6ebad431cb62043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:30:23.915575 8005 raft_consensus.cc:729] T 6b21f25189604ca495d4b4bc0adda37f P b580081cd50c4c80a6ebad431cb62043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b580081cd50c4c80a6ebad431cb62043, State: Initialized, Role: FOLLOWER
I20260929 20:30:23.916342 8005 consensus_queue.cc:260] T 6b21f25189604ca495d4b4bc0adda37f P b580081cd50c4c80a6ebad431cb62043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbda93b240994b278bf354743af462b2" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 43905 } } peers { permanent_uuid: "c7947c42ec424c058bfd54779eb4b8c4" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 43557 } } peers { permanent_uuid: "b580081cd50c4c80a6ebad431cb62043" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 34645 } }
I20260929 20:30:23.918368 7983 heartbeater.cc:503] Master 127.28.244.190:35649 was elected leader, sending a full tablet report...
I20260929 20:30:23.918752 8005 ts_tablet_manager.cc:1428] T 6b21f25189604ca495d4b4bc0adda37f P b580081cd50c4c80a6ebad431cb62043: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260929 20:30:23.920284 8003 ts_tablet_manager.cc:1428] T 6b21f25189604ca495d4b4bc0adda37f P dbda93b240994b278bf354743af462b2: Time spent starting tablet: real 0.018s user 0.016s sys 0.004s
W20260929 20:30:23.961475 7716 tablet.cc:2369] T 6b21f25189604ca495d4b4bc0adda37f P dbda93b240994b278bf354743af462b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:30:23.992450 7850 tablet.cc:2369] T 6b21f25189604ca495d4b4bc0adda37f P c7947c42ec424c058bfd54779eb4b8c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:30:24.028079 7984 tablet.cc:2369] T 6b21f25189604ca495d4b4bc0adda37f P b580081cd50c4c80a6ebad431cb62043: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:30:24.187995 8008 raft_consensus.cc:482] T 6b21f25189604ca495d4b4bc0adda37f P c7947c42ec424c058bfd54779eb4b8c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:30:24.188344 8008 raft_consensus.cc:504] T 6b21f25189604ca495d4b4bc0adda37f P c7947c42ec424c058bfd54779eb4b8c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbda93b240994b278bf354743af462b2" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 43905 } } peers { permanent_uuid: "c7947c42ec424c058bfd54779eb4b8c4" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 43557 } } peers { permanent_uuid: "b580081cd50c4c80a6ebad431cb62043" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 34645 } }
I20260929 20:30:24.190235 8008 leader_election.cc:288] T 6b21f25189604ca495d4b4bc0adda37f P c7947c42ec424c058bfd54779eb4b8c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbda93b240994b278bf354743af462b2 (127.28.244.129:43905), b580081cd50c4c80a6ebad431cb62043 (127.28.244.131:34645)
I20260929 20:30:24.197494 8001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:24.190795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:60305 (local address 127.28.244.131:34645)
0929 20:30:24.191025 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:24.191032 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:24.191879 (+ 847us) server_negotiation.cc:407] Connection header received
0929 20:30:24.192039 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:24.192044 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:24.192109 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:24.192190 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:30:24.192798 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:24.193694 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:24.195658 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:24.196509 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:24.196716 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:24.196799 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 20:30:24.197092 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 20:30:24.197225 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 20:30:24.197629 8011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:24.190620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:34645 (local address 127.28.244.130:60305)
0929 20:30:24.191725 (+ 1105us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:24.191741 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:24.191879 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:24.192223 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:24.192226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:24.192246 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:30:24.192626 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:24.192637 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:24.193832 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:24.193838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:24.195471 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:24.195486 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:24.196737 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:24.196744 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:24.196908 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:24.196933 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 20:30:24.197061 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 20:30:24.197217 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":999,"mutex_wait_us":45,"thread_start_us":604,"threads_started":1}
I20260929 20:30:24.197806 7997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:24.190836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:47981 (local address 127.28.244.129:43905)
0929 20:30:24.191031 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:24.191037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:24.191780 (+ 743us) server_negotiation.cc:407] Connection header received
0929 20:30:24.192389 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:24.192395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:24.192472 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:24.192542 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:30:24.193213 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:24.194190 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:24.196040 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:24.196816 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:24.197010 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:24.197095 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 20:30:24.197411 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 20:30:24.197563 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:30:24.198276 8012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:24.190553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:43905 (local address 127.28.244.130:47981)
0929 20:30:24.191650 (+ 1097us) negotiation.cc:107] Waiting for socket to connect
0929 20:30:24.191672 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:30:24.191879 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:30:24.192612 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:30:24.192617 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:30:24.192640 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:30:24.193071 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:24.193081 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:24.194341 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:24.194347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:24.195848 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:30:24.195869 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:30:24.197042 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:30:24.197048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:30:24.197235 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:24.197259 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 20:30:24.197379 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 20:30:24.197511 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":937,"mutex_wait_us":1,"thread_start_us":678,"threads_started":1}
I20260929 20:30:24.198827 7938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b21f25189604ca495d4b4bc0adda37f" candidate_uuid: "c7947c42ec424c058bfd54779eb4b8c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b580081cd50c4c80a6ebad431cb62043" is_pre_election: true
I20260929 20:30:24.199405 7938 raft_consensus.cc:2454] T 6b21f25189604ca495d4b4bc0adda37f P b580081cd50c4c80a6ebad431cb62043 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7947c42ec424c058bfd54779eb4b8c4 in term 0.
I20260929 20:30:24.199391 7670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b21f25189604ca495d4b4bc0adda37f" candidate_uuid: "c7947c42ec424c058bfd54779eb4b8c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbda93b240994b278bf354743af462b2" is_pre_election: true
I20260929 20:30:24.199980 7670 raft_consensus.cc:2454] T 6b21f25189604ca495d4b4bc0adda37f P dbda93b240994b278bf354743af462b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7947c42ec424c058bfd54779eb4b8c4 in term 0.
I20260929 20:30:24.200179 7739 leader_election.cc:302] T 6b21f25189604ca495d4b4bc0adda37f P c7947c42ec424c058bfd54779eb4b8c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b580081cd50c4c80a6ebad431cb62043, c7947c42ec424c058bfd54779eb4b8c4; no voters:
I20260929 20:30:24.200580 8008 raft_consensus.cc:2781] T 6b21f25189604ca495d4b4bc0adda37f P c7947c42ec424c058bfd54779eb4b8c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:30:24.200749 8008 raft_consensus.cc:482] T 6b21f25189604ca495d4b4bc0adda37f P c7947c42ec424c058bfd54779eb4b8c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:30:24.200871 8008 raft_consensus.cc:3037] T 6b21f25189604ca495d4b4bc0adda37f P c7947c42ec424c058bfd54779eb4b8c4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:30:24.203063 8008 raft_consensus.cc:504] T 6b21f25189604ca495d4b4bc0adda37f P c7947c42ec424c058bfd54779eb4b8c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbda93b240994b278bf354743af462b2" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 43905 } } peers { permanent_uuid: "c7947c42ec424c058bfd54779eb4b8c4" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 43557 } } peers { permanent_uuid: "b580081cd50c4c80a6ebad431cb62043" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 34645 } }
I20260929 20:30:24.203764 8008 leader_election.cc:288] T 6b21f25189604ca495d4b4bc0adda37f P c7947c42ec424c058bfd54779eb4b8c4 [CANDIDATE]: Term 1 election: Requested vote from peers dbda93b240994b278bf354743af462b2 (127.28.244.129:43905), b580081cd50c4c80a6ebad431cb62043 (127.28.244.131:34645)
I20260929 20:30:24.204231 7670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b21f25189604ca495d4b4bc0adda37f" candidate_uuid: "c7947c42ec424c058bfd54779eb4b8c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbda93b240994b278bf354743af462b2"
I20260929 20:30:24.204308 7938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b21f25189604ca495d4b4bc0adda37f" candidate_uuid: "c7947c42ec424c058bfd54779eb4b8c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b580081cd50c4c80a6ebad431cb62043"
I20260929 20:30:24.204432 7670 raft_consensus.cc:3037] T 6b21f25189604ca495d4b4bc0adda37f P dbda93b240994b278bf354743af462b2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:30:24.204552 7938 raft_consensus.cc:3037] T 6b21f25189604ca495d4b4bc0adda37f P b580081cd50c4c80a6ebad431cb62043 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:30:24.206674 7670 raft_consensus.cc:2454] T 6b21f25189604ca495d4b4bc0adda37f P dbda93b240994b278bf354743af462b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7947c42ec424c058bfd54779eb4b8c4 in term 1.
I20260929 20:30:24.206686 7938 raft_consensus.cc:2454] T 6b21f25189604ca495d4b4bc0adda37f P b580081cd50c4c80a6ebad431cb62043 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7947c42ec424c058bfd54779eb4b8c4 in term 1.
I20260929 20:30:24.207140 7737 leader_election.cc:302] T 6b21f25189604ca495d4b4bc0adda37f P c7947c42ec424c058bfd54779eb4b8c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7947c42ec424c058bfd54779eb4b8c4, dbda93b240994b278bf354743af462b2; no voters:
I20260929 20:30:24.207489 8008 raft_consensus.cc:2781] T 6b21f25189604ca495d4b4bc0adda37f P c7947c42ec424c058bfd54779eb4b8c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:30:24.208001 8008 raft_consensus.cc:686] T 6b21f25189604ca495d4b4bc0adda37f P c7947c42ec424c058bfd54779eb4b8c4 [term 1 LEADER]: Becoming Leader. State: Replica: c7947c42ec424c058bfd54779eb4b8c4, State: Running, Role: LEADER
I20260929 20:30:24.208482 8008 consensus_queue.cc:237] T 6b21f25189604ca495d4b4bc0adda37f P c7947c42ec424c058bfd54779eb4b8c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbda93b240994b278bf354743af462b2" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 43905 } } peers { permanent_uuid: "c7947c42ec424c058bfd54779eb4b8c4" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 43557 } } peers { permanent_uuid: "b580081cd50c4c80a6ebad431cb62043" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 34645 } }
I20260929 20:30:24.213339 7526 catalog_manager.cc:5495] T 6b21f25189604ca495d4b4bc0adda37f P c7947c42ec424c058bfd54779eb4b8c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to c7947c42ec424c058bfd54779eb4b8c4 (127.28.244.130). New cstate: current_term: 1 leader_uuid: "c7947c42ec424c058bfd54779eb4b8c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbda93b240994b278bf354743af462b2" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 43905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7947c42ec424c058bfd54779eb4b8c4" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 43557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b580081cd50c4c80a6ebad431cb62043" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 34645 } health_report { overall_health: UNKNOWN } } }
I20260929 20:30:24.245306 7998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:24.239181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36410 (local address 127.28.244.130:43557)
0929 20:30:24.239412 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:24.239417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:24.239492 (+ 75us) server_negotiation.cc:407] Connection header received
0929 20:30:24.239699 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:24.239703 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:24.239793 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:24.239977 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:30:24.240448 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:24.241230 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:24.243144 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:24.243569 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:24.243656 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:24.244212 (+ 556us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:30:24.244694 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:30:24.244798 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 20:30:24.244946 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 20:30:24.245014 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:30:24.254703 7938 raft_consensus.cc:1261] T 6b21f25189604ca495d4b4bc0adda37f P b580081cd50c4c80a6ebad431cb62043 [term 1 FOLLOWER]: Refusing update from remote peer c7947c42ec424c058bfd54779eb4b8c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:30:24.254706 7670 raft_consensus.cc:1261] T 6b21f25189604ca495d4b4bc0adda37f P dbda93b240994b278bf354743af462b2 [term 1 FOLLOWER]: Refusing update from remote peer c7947c42ec424c058bfd54779eb4b8c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:30:24.255694 8008 consensus_queue.cc:1035] T 6b21f25189604ca495d4b4bc0adda37f P c7947c42ec424c058bfd54779eb4b8c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbda93b240994b278bf354743af462b2" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 43905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:30:24.256075 8013 consensus_queue.cc:1035] T 6b21f25189604ca495d4b4bc0adda37f P c7947c42ec424c058bfd54779eb4b8c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b580081cd50c4c80a6ebad431cb62043" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 34645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:30:24.268574 8017 mvcc.cc:205] Tried to move back new op lower bound from 7334763824136273920 to 7334763823962107904. Current Snapshot: MvccSnapshot[applied={T|T < 7334763824136273920}]
I20260929 20:30:24.285457 7525 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:30:24.285782 7525 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:30:24.289721 7525 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1476027fabd48c7b1a4a97a92a0fbcf: Sending DeleteTablet for 3 replicas of tablet 6b21f25189604ca495d4b4bc0adda37f
I20260929 20:30:24.291129 7650 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b21f25189604ca495d4b4bc0adda37f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:30:24 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:35658
I20260929 20:30:24.291153 7918 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b21f25189604ca495d4b4bc0adda37f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:30:24 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:35214
I20260929 20:30:24.291898 7784 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b21f25189604ca495d4b4bc0adda37f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:30:24 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:36408
I20260929 20:30:24.292850 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 7587
I20260929 20:30:24.296478 8027 tablet_replica.cc:330] T 6b21f25189604ca495d4b4bc0adda37f P c7947c42ec424c058bfd54779eb4b8c4: stopping tablet replica
I20260929 20:30:24.297170 8027 raft_consensus.cc:2229] T 6b21f25189604ca495d4b4bc0adda37f P c7947c42ec424c058bfd54779eb4b8c4 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:30:24.298158 8027 raft_consensus.cc:2258] T 6b21f25189604ca495d4b4bc0adda37f P c7947c42ec424c058bfd54779eb4b8c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:30:24.298868 8026 tablet_replica.cc:330] T 6b21f25189604ca495d4b4bc0adda37f P dbda93b240994b278bf354743af462b2: stopping tablet replica
I20260929 20:30:24.299122 8025 tablet_replica.cc:330] T 6b21f25189604ca495d4b4bc0adda37f P b580081cd50c4c80a6ebad431cb62043: stopping tablet replica
I20260929 20:30:24.299727 8026 raft_consensus.cc:2229] T 6b21f25189604ca495d4b4bc0adda37f P dbda93b240994b278bf354743af462b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:30:24.299716 8025 raft_consensus.cc:2229] T 6b21f25189604ca495d4b4bc0adda37f P b580081cd50c4c80a6ebad431cb62043 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:30:24.300416 8026 raft_consensus.cc:2258] T 6b21f25189604ca495d4b4bc0adda37f P dbda93b240994b278bf354743af462b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:30:24.300452 8025 raft_consensus.cc:2258] T 6b21f25189604ca495d4b4bc0adda37f P b580081cd50c4c80a6ebad431cb62043 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:30:24.302620 8026 ts_tablet_manager.cc:1905] T 6b21f25189604ca495d4b4bc0adda37f P dbda93b240994b278bf354743af462b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:30:24.302628 8027 ts_tablet_manager.cc:1905] T 6b21f25189604ca495d4b4bc0adda37f P c7947c42ec424c058bfd54779eb4b8c4: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 20:30:24 dist-test-slave-p29l krb5kdc[7481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.129@KRBTEST.COM
I20260929 20:30:24.310261 8026 ts_tablet_manager.cc:1918] T 6b21f25189604ca495d4b4bc0adda37f P dbda93b240994b278bf354743af462b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:30:24.310331 8025 ts_tablet_manager.cc:1905] T 6b21f25189604ca495d4b4bc0adda37f P b580081cd50c4c80a6ebad431cb62043: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:30:24.310561 8026 log.cc:1196] T 6b21f25189604ca495d4b4bc0adda37f P dbda93b240994b278bf354743af462b2: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-0/wal/wals/6b21f25189604ca495d4b4bc0adda37f
I20260929 20:30:24.311146 8026 ts_tablet_manager.cc:1939] T 6b21f25189604ca495d4b4bc0adda37f P dbda93b240994b278bf354743af462b2: Deleting consensus metadata
I20260929 20:30:24.311483 8027 ts_tablet_manager.cc:1918] T 6b21f25189604ca495d4b4bc0adda37f P c7947c42ec424c058bfd54779eb4b8c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:30:24.311666 8027 log.cc:1196] T 6b21f25189604ca495d4b4bc0adda37f P c7947c42ec424c058bfd54779eb4b8c4: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-1/wal/wals/6b21f25189604ca495d4b4bc0adda37f
I20260929 20:30:24.312234 8027 ts_tablet_manager.cc:1939] T 6b21f25189604ca495d4b4bc0adda37f P c7947c42ec424c058bfd54779eb4b8c4: Deleting consensus metadata
I20260929 20:30:24.313557 7510 catalog_manager.cc:4841] TS dbda93b240994b278bf354743af462b2 (127.28.244.129:43905): tablet 6b21f25189604ca495d4b4bc0adda37f (table test-table [id=febe1082ea0b46b4bf87cf45fb47fa68]) successfully deleted
I20260929 20:30:24.313946 7512 catalog_manager.cc:4841] TS c7947c42ec424c058bfd54779eb4b8c4 (127.28.244.130:43557): tablet 6b21f25189604ca495d4b4bc0adda37f (table test-table [id=febe1082ea0b46b4bf87cf45fb47fa68]) successfully deleted
I20260929 20:30:24.317624 8025 ts_tablet_manager.cc:1918] T 6b21f25189604ca495d4b4bc0adda37f P b580081cd50c4c80a6ebad431cb62043: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:30:24.317808 8025 log.cc:1196] T 6b21f25189604ca495d4b4bc0adda37f P b580081cd50c4c80a6ebad431cb62043: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713721201454-29650-0/minicluster-data/ts-2/wal/wals/6b21f25189604ca495d4b4bc0adda37f
I20260929 20:30:24.318351 8025 ts_tablet_manager.cc:1939] T 6b21f25189604ca495d4b4bc0adda37f P b580081cd50c4c80a6ebad431cb62043: Deleting consensus metadata
I20260929 20:30:24.320358 7512 catalog_manager.cc:4841] TS b580081cd50c4c80a6ebad431cb62043 (127.28.244.131:34645): tablet 6b21f25189604ca495d4b4bc0adda37f (table test-table [id=febe1082ea0b46b4bf87cf45fb47fa68]) successfully deleted
I20260929 20:30:24.321503 7997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:24.294207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35674 (local address 127.28.244.129:43905)
0929 20:30:24.294384 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:24.294390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:24.296898 (+ 2508us) server_negotiation.cc:407] Connection header received
0929 20:30:24.297132 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:24.297140 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:24.297229 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:24.297336 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:24.300698 (+ 3362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:24.301924 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:24.303862 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:24.304556 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:24.304667 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:24.315110 (+ 10443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:30:24.315122 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:30:24.315137 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:30:24.315173 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:30:24.318316 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:30:24.319173 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:30:24.319181 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:30:24.319183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:30:24.319318 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:30:24.319763 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:30:24.319767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:30:24.319769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:30:24.320327 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:30:24.320675 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0929 20:30:24.320992 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 20:30:24.321175 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 20:30:24.322184 7710 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:30:24.431969 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 7587
I20260929 20:30:24.450867 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 7721
Sep 29 20:30:24 dist-test-slave-p29l krb5kdc[7481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.130@KRBTEST.COM
I20260929 20:30:24.468276 7998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:24.451839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36420 (local address 127.28.244.130:43557)
0929 20:30:24.452070 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:24.452081 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:24.452102 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:30:24.452170 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:24.452176 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:24.452282 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:24.452412 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:24.453767 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:24.454990 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:24.456326 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:24.457017 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:24.457147 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:24.461253 (+ 4106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:30:24.461275 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:30:24.461302 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:30:24.461361 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:30:24.465314 (+ 3953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:30:24.466052 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:30:24.466062 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:30:24.466066 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:30:24.466240 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:30:24.466588 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:30:24.466593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:30:24.466597 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:30:24.467074 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:30:24.467379 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0929 20:30:24.467578 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 20:30:24.467841 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:30:24.469142 7844 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:30:24.584051 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 7721
I20260929 20:30:24.605439 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 7855
Sep 29 20:30:24 dist-test-slave-p29l krb5kdc[7481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.131@KRBTEST.COM
I20260929 20:30:24.620353 8001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:30:24.606284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35222 (local address 127.28.244.131:34645)
0929 20:30:24.606492 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 20:30:24.606497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:30:24.606517 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:30:24.606600 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:30:24.606605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:30:24.606683 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:30:24.606774 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:30:24.608219 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:24.608946 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:24.610198 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:30:24.610595 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:30:24.610693 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:30:24.615218 (+ 4525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:30:24.615232 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:30:24.615249 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:30:24.615287 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:30:24.617921 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:30:24.618535 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:30:24.618542 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:30:24.618544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:30:24.618685 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:30:24.618996 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:30:24.618999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:30:24.619001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:30:24.619375 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:30:24.619596 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 20:30:24.619899 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 20:30:24.620089 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:30:24.620971 7978 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:30:24.709367 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 7855
I20260929 20:30:24.727643 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 7494
I20260929 20:30:24.728884 7556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:30:24.839293 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 7494
2026-09-29T20:30:24Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4654 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:30:24 dist-test-slave-p29l krb5kdc[8035](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:30:24 dist-test-slave-p29l krb5kdc[8035](info): set up 2 sockets
Sep 29 20:30:24 dist-test-slave-p29l krb5kdc[8035](info): commencing operation
krb5kdc: starting...
W20260929 20:30:25.992936 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:30:26 dist-test-slave-p29l krb5kdc[8035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:30:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:30:26Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.244.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.244.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.244.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/krb5kdc/rangeradmin_127.28.244.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.244.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/krb5kdc/rangeradmin_127.28.244.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.244.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.244.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.244.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/krb5kdc/rangerlookup_127.28.244.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.244.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/krb5kdc/rangerlookup_127.28.244.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.244.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.244.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.244.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/krb5kdc/HTTP_127.28.244.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.244.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/krb5kdc/HTTP_127.28.244.148@KRBTEST.COM.keytab.
I20260929 20:30:26.338337 29650 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskNrwVME/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/postgres -l logfile start
2026-09-29 20:30:29.056 UTC [8065] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 20:30:29.056 UTC [8065] LOG: listening on IPv4 address "127.28.244.148", port 48983
2026-09-29 20:30:29.061 UTC [8065] LOG: listening on Unix socket "/tmp/.s.PGSQL.48983"
2026-09-29 20:30:29.084 UTC [8068] LOG: database system was shut down at 2026-09-29 20:30:27 UTC
2026-09-29 20:30:29.088 UTC [8065] LOG: database system is ready to accept connections
I20260929 20:30:30.079878 29650 mini_postgres.cc:96] Postgres bound to 48983
2026-09-29 20:30:30.098 UTC [8076] FATAL: database "slave" does not exist
127.28.244.148:48983 - accepting connections
I20260929 20:30:30.099272 29650 mini_ranger.cc:161] Starting Ranger...
I20260929 20:30:30.126621 29650 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 20:30:30.649379 29650 mini_ranger.cc:87] Created ranger Postgres database
I20260929 20:30:30.649593 29650 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ranger-admin
2026-09-29 20:30:31,080 [I] DB FLAVOR :POSTGRES
2026-09-29 20:30:31,080 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 20:30:31,080 [I] Checking connection..
2026-09-29 20:30:31,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:48983/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 20:30:31,357 [I] Checking connection passed.
2026-09-29 20:30:31,357 [I] --------- Verifying version history table ---------
2026-09-29 20:30:31,357 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:48983/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 20:30:31,650 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 20:30:31,650 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:48983/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 20:30:31,923 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 20:30:31,924 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 20:30:31,924 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:48983/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 20:30:32.162 UTC [8140] WARNING: there is no transaction in progress
2026-09-29 20:30:32,169 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 20:30:32,169 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:48983/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 20:30:32,412 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 20:30:32,413 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 20:30:32,413 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:48983/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 20:30:32,665 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:48983/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 20:30:32,896 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:48983/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-p29l.c.gcp-upstream.internal','N') ;"
2026-09-29 20:30:33,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:48983/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 20:30:33,419 [I] Table x_portal_user does not exist in database ranger
2026-09-29 20:30:33,419 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:48983/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 20:30:33,688 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 20:30:33,689 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:48983/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 20:30:33,930 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:48983/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 20:30:34,188 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 20:30:34,188 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:48983/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 20:30:35.209 UTC [8248] WARNING: there is no transaction in progress
2026-09-29 20:30:35.226 UTC [8248] WARNING: there is no transaction in progress
2026-09-29 20:30:35.244 UTC [8248] WARNING: there is no transaction in progress
2026-09-29 20:30:35.261 UTC [8248] WARNING: there is no transaction in progress
2026-09-29 20:30:35.279 UTC [8248] WARNING: there is no transaction in progress
2026-09-29 20:30:35.298 UTC [8248] WARNING: there is no transaction in progress
2026-09-29 20:30:35.356 UTC [8248] WARNING: there is no transaction in progress
2026-09-29 20:30:35.363 UTC [8248] WARNING: there is no transaction in progress
2026-09-29 20:30:35.379 UTC [8248] WARNING: there is no transaction in progress
2026-09-29 20:30:35.390 UTC [8248] WARNING: there is no transaction in progress
2026-09-29 20:30:35.401 UTC [8248] WARNING: there is no transaction in progress
2026-09-29 20:30:35.416 UTC [8248] WARNING: there is no transaction in progress
2026-09-29 20:30:35.426 UTC [8248] WARNING: there is no transaction in progress
2026-09-29 20:30:35.433 UTC [8248] WARNING: there is no transaction in progress
2026-09-29 20:30:35.440 UTC [8248] WARNING: there is no transaction in progress
2026-09-29 20:30:35.448 UTC [8248] WARNING: there is no transaction in progress
2026-09-29 20:30:36,155 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 20:30:36,156 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:48983/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 20:30:36,400 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 20:30:36,400 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:48983/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 20:30:36,669 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 20:30:36,670 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:48983/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 20:30:36,926 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 20:30:36,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:48983/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 20:30:37,188 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:48983/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 20:30:37,426 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:48983/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-p29l.c.gcp-upstream.internal';"
2026-09-29 20:30:37,688 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 20:30:37,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:48983/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 20:30:37,935 [I] DB_PATCHES have already been applied
I20260929 20:30:37.941572 29650 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ranger-admin:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskNrwVME/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 20:30:37.941670 29650 mini_ranger.cc:193] Using host: 127.28.244.148
I20260929 20:30:37.957232 29650 mini_ranger.cc:227] Ranger admin URL: http://127.28.244.148:40395
Sep 29, 2026 8:30:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 8:30:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 8:30:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 8:30:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 8:30:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 8:30:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 8:30:38 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 20:30:39.137245 29650 mini_ranger.cc:160] Time spent starting Ranger: real 9.038s user 0.003s sys 0.041s
Sep 29, 2026 8:30:59 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 8:30:59 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 8:30:59 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 8:30:59 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 8:30:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 8:30:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 8:30:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 8:30:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 8:30:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 8:30:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 8:30:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 8:30:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 8:30:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 8:30:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 8:30:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 8:30:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 8:30:59 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 8:31:00 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 20:31:01.165607 29650 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.244.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.244.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.244.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/krb5kdc/rangerkms_127.28.244.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.244.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/krb5kdc/rangerkms_127.28.244.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.244.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.244.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.244.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/krb5kdc/HTTP_127.28.244.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.244.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/krb5kdc/HTTP_127.28.244.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 20:31:01 dist-test-slave-p29l krb5kdc[8035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713861, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 20:31:01.365602 29650 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 20:31:01.388538 29650 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 20:31:02.306533 29650 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 20:31:02.306715 29650 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ranger-kms
I20260929 20:31:02.306866 29650 mini_ranger_kms.cc:227] Using postgres at 127.28.244.148:48983
2026-09-29 20:31:03,010 [I] DB FLAVOR :POSTGRES
2026-09-29 20:31:03,011 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 20:31:03,011 [I] Checking connection
2026-09-29 20:31:03,011 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:48983/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 20:31:03,262 [I] connection success
2026-09-29 20:31:03,263 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 20:31:03,263 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 20:31:03,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:48983/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 20:31:03,502 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 20:31:03,503 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 20:31:03,503 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 20:31:03,503 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:48983/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 20:31:03,795 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 20:31:04.341979 29650 mini_ranger_kms.cc:310] Created kms service
I20260929 20:31:04.558884 29650 mini_ranger_kms.cc:326] Created kudu user
I20260929 20:31:04.609910 29650 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 20:31:04 dist-test-slave-p29l krb5kdc[8035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713861, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.244.148@KRBTEST.COM
I20260929 20:31:05.073863 29650 mini_ranger_kms.cc:384] Added ranger policy
I20260929 20:31:05.074189 29650 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ranger-kms:/tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 20:31:05.074259 29650 mini_ranger_kms.cc:240] Using host: 127.28.244.148
I20260929 20:31:05.094120 29650 mini_ranger_kms.cc:276] Ranger KMS PID: 8429
I20260929 20:31:05.094339 29650 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.244.148:56181
Sep 29, 2026 8:31:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 8:31:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 8:31:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 8:31:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 8:31:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 8:31:07 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 20:31:07.426465 29650 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.061s user 0.112s sys 0.382s
I20260929 20:31:07.426741 29650 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 20:31:07.426965 29650 mini_ranger_kms.cc:401] 127.28.244.148:56181/kms/v1/keys
Sep 29 20:31:07 dist-test-slave-p29l krb5kdc[8035](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790713867, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.244.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 20:31:08 dist-test-slave-p29l krb5kdc[8035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713867, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.244.148@KRBTEST.COM for HTTP/127.28.244.148@KRBTEST.COM
Sep 29 20:31:08 dist-test-slave-p29l krb5kdc[8035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713867, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.244.148@KRBTEST.COM for HTTP/127.28.244.148@KRBTEST.COM
Sep 29 20:31:10 dist-test-slave-p29l krb5kdc[8035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.244.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
I20260929 20:31:11.060276 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:38235
--webserver_interface=127.28.244.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:43161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:38235
--ranger_config_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.244.148:56181/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:31:11.418751 8479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:31:11.419070 8479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:31:11.419157 8479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:31:11.426662 8479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:31:11.426783 8479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:31:11.426827 8479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:31:11.426875 8479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:31:11.426924 8479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:31:11.437688 8479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:43161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.244.148:56181/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:38235
--ranger_config_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:38235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:31:11.439128 8479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:31:11.440497 8479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:31:11.453754 8484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:31:11.453688 8487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:31:11.453729 8485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:31:11.455201 8479 server_base.cc:1034] running on GCE node
I20260929 20:31:11.456171 8479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:31:11.457796 8479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:31:11.459030 8479 hybrid_clock.cc:648] HybridClock initialized: now 1790713871458988 us; error 38 us; skew 500 ppm
I20260929 20:31:11.459455 8479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:31:11 dist-test-slave-p29l krb5kdc[8035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713871, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:31:11.467912 8479 init.cc:377] Logged in from keytab as kudu/127.28.244.190@KRBTEST.COM (short username kudu)
I20260929 20:31:11.469635 8479 webserver.cc:460] Webserver started at http://127.28.244.190:42673/ using document root <none> and password file <none>
I20260929 20:31:11.470378 8479 fs_manager.cc:359] Metadata directory not provided
I20260929 20:31:11.470518 8479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:31:11.470875 8479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 20:31:11 dist-test-slave-p29l krb5kdc[8035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713871, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for HTTP/127.28.244.148@KRBTEST.COM
I20260929 20:31:11.533437 8479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "d7229fd9a18046b98cee658a29ca7936"
format_stamp: "Formatted at 2026-09-29 20:31:11 on dist-test-slave-p29l"
server_key: "607bf0fad24ffda13272c55f8a784ea1"
server_key_iv: "960c350b2ec07d6bf96ee6f0f97e7773"
server_key_version: "kuduclusterkey@0"
I20260929 20:31:11.534662 8479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "d7229fd9a18046b98cee658a29ca7936"
format_stamp: "Formatted at 2026-09-29 20:31:11 on dist-test-slave-p29l"
server_key: "607bf0fad24ffda13272c55f8a784ea1"
server_key_iv: "960c350b2ec07d6bf96ee6f0f97e7773"
server_key_version: "kuduclusterkey@0"
I20260929 20:31:11.542511 8479 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260929 20:31:11.561415 8494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:31:11.563335 8479 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 20:31:11.563586 8479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "d7229fd9a18046b98cee658a29ca7936"
format_stamp: "Formatted at 2026-09-29 20:31:11 on dist-test-slave-p29l"
server_key: "607bf0fad24ffda13272c55f8a784ea1"
server_key_iv: "960c350b2ec07d6bf96ee6f0f97e7773"
server_key_version: "kuduclusterkey@0"
I20260929 20:31:11.563890 8479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:31:11.581880 8479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:31:11.586587 8479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:31:11.586930 8479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:31:11.608091 8479 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:38235
I20260929 20:31:11.608101 8545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:38235 every 8 connection(s)
I20260929 20:31:11.609747 8479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:31:11.610383 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 8479
I20260929 20:31:11.610626 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:31:11.611682 29650 external_mini_cluster.cc:1428] Setting key 4a51dad0f865d78b1858ef75a052648b
I20260929 20:31:11.615459 8546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 20:31:11 dist-test-slave-p29l krb5kdc[8035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:31:11.629892 8546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7229fd9a18046b98cee658a29ca7936: Bootstrap starting.
I20260929 20:31:11.633512 8546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7229fd9a18046b98cee658a29ca7936: Neither blocks nor log segments found. Creating new log.
I20260929 20:31:11.634684 8546 log.cc:824] T 00000000000000000000000000000000 P d7229fd9a18046b98cee658a29ca7936: Log is configured to *not* fsync() on all Append() calls
I20260929 20:31:11.638794 8549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:31:11.615406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51780 (local address 127.28.244.190:38235)
0929 20:31:11.616373 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0929 20:31:11.616385 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:31:11.616432 (+ 47us) server_negotiation.cc:407] Connection header received
0929 20:31:11.618120 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:31:11.618143 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:31:11.618756 (+ 613us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:31:11.619219 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:31:11.621130 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:31:11.622951 (+ 1821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:31:11.624845 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:31:11.625624 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:31:11.625786 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:31:11.630678 (+ 4892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:31:11.630689 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:31:11.630706 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:31:11.630751 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:31:11.634287 (+ 3536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:31:11.635194 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:31:11.635206 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:31:11.635213 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:31:11.635391 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:31:11.635885 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:31:11.635889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:31:11.635891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:31:11.636438 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:31:11.636746 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0929 20:31:11.637313 (+ 567us) server_negotiation.cc:299] Negotiation successful
0929 20:31:11.637755 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":736,"thread_start_us":514,"threads_started":1}
I20260929 20:31:11.639050 8546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7229fd9a18046b98cee658a29ca7936: No bootstrap required, opened a new log
I20260929 20:31:11.647764 8546 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7229fd9a18046b98cee658a29ca7936 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7229fd9a18046b98cee658a29ca7936" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 38235 } }
I20260929 20:31:11.648384 8546 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7229fd9a18046b98cee658a29ca7936 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:31:11.648512 8546 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7229fd9a18046b98cee658a29ca7936 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7229fd9a18046b98cee658a29ca7936, State: Initialized, Role: FOLLOWER
I20260929 20:31:11.649255 8546 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7229fd9a18046b98cee658a29ca7936 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7229fd9a18046b98cee658a29ca7936" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 38235 } }
I20260929 20:31:11.649518 8546 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7229fd9a18046b98cee658a29ca7936 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:31:11.649667 8546 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7229fd9a18046b98cee658a29ca7936 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:31:11.649858 8546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7229fd9a18046b98cee658a29ca7936 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:31:11.652251 8546 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7229fd9a18046b98cee658a29ca7936 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7229fd9a18046b98cee658a29ca7936" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 38235 } }
I20260929 20:31:11.652865 8546 leader_election.cc:302] T 00000000000000000000000000000000 P d7229fd9a18046b98cee658a29ca7936 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7229fd9a18046b98cee658a29ca7936; no voters:
I20260929 20:31:11.653492 8546 leader_election.cc:288] T 00000000000000000000000000000000 P d7229fd9a18046b98cee658a29ca7936 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:31:11.653717 8551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7229fd9a18046b98cee658a29ca7936 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:31:11.654521 8551 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7229fd9a18046b98cee658a29ca7936 [term 1 LEADER]: Becoming Leader. State: Replica: d7229fd9a18046b98cee658a29ca7936, State: Running, Role: LEADER
I20260929 20:31:11.655098 8551 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7229fd9a18046b98cee658a29ca7936 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7229fd9a18046b98cee658a29ca7936" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 38235 } }
I20260929 20:31:11.655997 8546 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7229fd9a18046b98cee658a29ca7936 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:31:11.658152 8553 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7229fd9a18046b98cee658a29ca7936 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7229fd9a18046b98cee658a29ca7936. Latest consensus state: current_term: 1 leader_uuid: "d7229fd9a18046b98cee658a29ca7936" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7229fd9a18046b98cee658a29ca7936" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 38235 } } }
I20260929 20:31:11.659161 8553 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7229fd9a18046b98cee658a29ca7936 [sys.catalog]: This master's current role is: LEADER
I20260929 20:31:11.658862 8552 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7229fd9a18046b98cee658a29ca7936 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7229fd9a18046b98cee658a29ca7936" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7229fd9a18046b98cee658a29ca7936" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 38235 } } }
I20260929 20:31:11.660180 8552 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7229fd9a18046b98cee658a29ca7936 [sys.catalog]: This master's current role is: LEADER
I20260929 20:31:11.662864 8559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:31:11.669176 8559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:31:11.670182 8546 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 20:31:11.687871 8559 catalog_manager.cc:1347] Generated new cluster ID: b1d4fce32b2c46e188c2b1b375a4417e
I20260929 20:31:11.688122 8559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:31:11.702390 8559 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:31:11.704686 8559 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:31:11.715276 8559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7229fd9a18046b98cee658a29ca7936: Generated new TSK 0
I20260929 20:31:11.717077 8559 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.244.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab.
Entry for principal kudu/127.28.244.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab.
I20260929 20:31:12.511353 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.244.148:56181/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.129:0
--local_ip_for_outbound_sockets=127.28.244.129
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:38235
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:43161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:31:12 dist-test-slave-p29l krb5kdc[8035](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790713872, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 20:31:13.036962 8591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:31:13.037307 8591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:31:13.037492 8591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:31:13.046604 8591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:31:13.046761 8591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:31:13.046970 8591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.129
I20260929 20:31:13.063199 8591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:43161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.244.148:56181/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:38235
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.129
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:31:13.065414 8591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:31:13.067525 8591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:31:13.087409 8601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:31:13.087296 8602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:31:13.095412 8604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:31:13.098090 8591 server_base.cc:1034] running on GCE node
I20260929 20:31:13.099076 8591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:31:13.100598 8591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:31:13.101999 8591 hybrid_clock.cc:648] HybridClock initialized: now 1790713873101875 us; error 110 us; skew 500 ppm
I20260929 20:31:13.102476 8591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:31:13 dist-test-slave-p29l krb5kdc[8035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713873, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:31:13.107764 8591 init.cc:377] Logged in from keytab as kudu/127.28.244.129@KRBTEST.COM (short username kudu)
I20260929 20:31:13.120383 8591 webserver.cc:460] Webserver started at http://127.28.244.129:35873/ using document root <none> and password file <none>
I20260929 20:31:13.121742 8591 fs_manager.cc:359] Metadata directory not provided
I20260929 20:31:13.122042 8591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:31:13.122570 8591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 20:31:13 dist-test-slave-p29l krb5kdc[8035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713873, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for HTTP/127.28.244.148@KRBTEST.COM
I20260929 20:31:13.149840 8591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-0/data/instance:
uuid: "b92e4ee58b004430801ef2831b495dd9"
format_stamp: "Formatted at 2026-09-29 20:31:13 on dist-test-slave-p29l"
server_key: "6fcdc8069cd1938ce9e2f998202ecc83"
server_key_iv: "3db74488090f33ae299413897d83dc23"
server_key_version: "kuduclusterkey@0"
I20260929 20:31:13.152367 8591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance:
uuid: "b92e4ee58b004430801ef2831b495dd9"
format_stamp: "Formatted at 2026-09-29 20:31:13 on dist-test-slave-p29l"
server_key: "6fcdc8069cd1938ce9e2f998202ecc83"
server_key_iv: "3db74488090f33ae299413897d83dc23"
server_key_version: "kuduclusterkey@0"
I20260929 20:31:13.167925 8591 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.010s sys 0.004s
I20260929 20:31:13.200142 8611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:31:13.202553 8591 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.004s
I20260929 20:31:13.202940 8591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-0/wal
uuid: "b92e4ee58b004430801ef2831b495dd9"
format_stamp: "Formatted at 2026-09-29 20:31:13 on dist-test-slave-p29l"
server_key: "6fcdc8069cd1938ce9e2f998202ecc83"
server_key_iv: "3db74488090f33ae299413897d83dc23"
server_key_version: "kuduclusterkey@0"
I20260929 20:31:13.203365 8591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:31:13.251856 8591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:31:13.258391 8591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:31:13.258895 8591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:31:13.262300 8591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
Sep 29 20:31:13 dist-test-slave-p29l krb5kdc[8035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713872, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for HTTP/127.28.244.148@KRBTEST.COM
I20260929 20:31:13.264882 8591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:31:13.265013 8591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:31:13.265190 8591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:31:13.265280 8591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:31:13.368124 8591 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.129:36409
I20260929 20:31:13.368180 8723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.129:36409 every 8 connection(s)
I20260929 20:31:13.369797 8591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
I20260929 20:31:13.376936 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 8591
I20260929 20:31:13.377202 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance
I20260929 20:31:13.378175 29650 external_mini_cluster.cc:1428] Setting key 45e7e22cb6fbb9a6c3c8d3b20a04e6a9
Sep 29 20:31:13 dist-test-slave-p29l krb5kdc[8035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713873, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:31:13.396045 8726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:31:13.373023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:36827 (local address 127.28.244.190:38235)
0929 20:31:13.373554 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0929 20:31:13.373559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:31:13.374961 (+ 1402us) server_negotiation.cc:407] Connection header received
0929 20:31:13.377769 (+ 2808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:31:13.377776 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:31:13.377890 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:31:13.378029 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:31:13.380297 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:31:13.381159 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:31:13.383103 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:31:13.383588 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:31:13.383751 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:31:13.387752 (+ 4001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:31:13.387758 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:31:13.387762 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:31:13.387798 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:31:13.391282 (+ 3484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:31:13.392581 (+ 1299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:31:13.392588 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:31:13.392591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:31:13.392769 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:31:13.393467 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:31:13.393472 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:31:13.393475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:31:13.393858 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:31:13.394015 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 20:31:13.395187 (+ 1172us) server_negotiation.cc:299] Negotiation successful
0929 20:31:13.395507 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":300,"threads_started":1}
I20260929 20:31:13.397508 8727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:31:13.373844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:38235 (local address 127.28.244.129:36827)
0929 20:31:13.374753 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0929 20:31:13.374798 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 20:31:13.377307 (+ 2509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:31:13.378399 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:31:13.378415 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:31:13.379120 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:31:13.380088 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:31:13.380113 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:31:13.381374 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:31:13.381378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:31:13.382890 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:31:13.382937 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:31:13.383899 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:31:13.383909 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:31:13.384253 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:31:13.385121 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:31:13.385149 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:31:13.387436 (+ 2287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:31:13.391503 (+ 4067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:31:13.391519 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:31:13.391544 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:31:13.392365 (+ 821us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:31:13.392918 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:31:13.392927 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:31:13.392934 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:31:13.393293 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:31:13.394036 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:31:13.394050 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:31:13.394636 (+ 586us) client_negotiation.cc:769] Sending connection context
0929 20:31:13.395016 (+ 380us) client_negotiation.cc:240] Negotiation successful
0929 20:31:13.395702 (+ 686us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":347,"threads_started":1}
I20260929 20:31:13.399695 8724 heartbeater.cc:347] Connected to a master server at 127.28.244.190:38235
I20260929 20:31:13.400162 8724 heartbeater.cc:464] Registering TS with master...
I20260929 20:31:13.401206 8724 heartbeater.cc:511] Master 127.28.244.190:38235 requested a full tablet report, sending...
I20260929 20:31:13.404031 8511 ts_manager.cc:194] Registered new tserver with Master: b92e4ee58b004430801ef2831b495dd9 (127.28.244.129:36409)
I20260929 20:31:13.406219 8511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.129@KRBTEST.COM'} at 127.28.244.129:36827
WARNING: no policy specified for kudu/127.28.244.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab.
Entry for principal kudu/127.28.244.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab.
I20260929 20:31:13.449982 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.244.148:56181/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.130:0
--local_ip_for_outbound_sockets=127.28.244.130
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:38235
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:43161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:31:13 dist-test-slave-p29l krb5kdc[8035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713872, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for HTTP/127.28.244.148@KRBTEST.COM
W20260929 20:31:13.941977 8730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:31:13.942343 8730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:31:13.942514 8730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:31:13.950853 8730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:31:13.951033 8730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:31:13.951259 8730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.130
I20260929 20:31:13.966629 8730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:43161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.244.148:56181/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:38235
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.130
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:31:13.968853 8730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:31:13.970825 8730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:31:13.988328 8744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:31:13.988323 8745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:31:13.989558 8730 server_base.cc:1034] running on GCE node
W20260929 20:31:13.989761 8747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:31:13.990537 8730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:31:13.991739 8730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:31:13.992988 8730 hybrid_clock.cc:648] HybridClock initialized: now 1790713873992940 us; error 45 us; skew 500 ppm
I20260929 20:31:13.993350 8730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:31:13 dist-test-slave-p29l krb5kdc[8035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713873, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:31:13.997265 8730 init.cc:377] Logged in from keytab as kudu/127.28.244.130@KRBTEST.COM (short username kudu)
I20260929 20:31:13.999320 8730 webserver.cc:460] Webserver started at http://127.28.244.130:45239/ using document root <none> and password file <none>
I20260929 20:31:14.000371 8730 fs_manager.cc:359] Metadata directory not provided
I20260929 20:31:14.000542 8730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:31:14.000936 8730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 20:31:14 dist-test-slave-p29l krb5kdc[8035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713873, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for HTTP/127.28.244.148@KRBTEST.COM
I20260929 20:31:14.021381 8730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-1/data/instance:
uuid: "a00ef07dc34d43ba8028808bd0a9d464"
format_stamp: "Formatted at 2026-09-29 20:31:14 on dist-test-slave-p29l"
server_key: "ad98e97e9f2bbff971dd31a0942ed8ce"
server_key_iv: "d8b39cb4b4baf9ebd4e70872b17c481d"
server_key_version: "kuduclusterkey@0"
I20260929 20:31:14.022461 8730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance:
uuid: "a00ef07dc34d43ba8028808bd0a9d464"
format_stamp: "Formatted at 2026-09-29 20:31:14 on dist-test-slave-p29l"
server_key: "ad98e97e9f2bbff971dd31a0942ed8ce"
server_key_iv: "d8b39cb4b4baf9ebd4e70872b17c481d"
server_key_version: "kuduclusterkey@0"
I20260929 20:31:14.030716 8730 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.001s sys 0.008s
I20260929 20:31:14.061470 8754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:31:14.063588 8730 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.007s sys 0.009s
I20260929 20:31:14.063879 8730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-1/wal
uuid: "a00ef07dc34d43ba8028808bd0a9d464"
format_stamp: "Formatted at 2026-09-29 20:31:14 on dist-test-slave-p29l"
server_key: "ad98e97e9f2bbff971dd31a0942ed8ce"
server_key_iv: "d8b39cb4b4baf9ebd4e70872b17c481d"
server_key_version: "kuduclusterkey@0"
I20260929 20:31:14.064194 8730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:31:14.083119 8730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:31:14.088151 8730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:31:14.088531 8730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:31:14.089771 8730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:31:14.091528 8730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:31:14.091645 8730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:31:14.091851 8730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:31:14.091948 8730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260929 20:31:14.135604 8730 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.130:34049
I20260929 20:31:14.135625 8866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.130:34049 every 8 connection(s)
I20260929 20:31:14.137310 8730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
I20260929 20:31:14.141184 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 8730
I20260929 20:31:14.141367 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance
I20260929 20:31:14.141961 29650 external_mini_cluster.cc:1428] Setting key 87b2c354b50195d35bf71b8abe04f2e4
Sep 29 20:31:14 dist-test-slave-p29l krb5kdc[8035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713873, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:31:14.167055 8869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:31:14.140520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:53163 (local address 127.28.244.190:38235)
0929 20:31:14.141031 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0929 20:31:14.141038 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:31:14.142492 (+ 1454us) server_negotiation.cc:407] Connection header received
0929 20:31:14.145361 (+ 2869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:31:14.145370 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:31:14.145497 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:31:14.145591 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:31:14.148666 (+ 3075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:31:14.150124 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:31:14.153171 (+ 3047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:31:14.153905 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:31:14.154037 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:31:14.159681 (+ 5644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:31:14.159687 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:31:14.159691 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:31:14.159744 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:31:14.163109 (+ 3365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:31:14.164226 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:31:14.164232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:31:14.164234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:31:14.164371 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:31:14.164904 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:31:14.164908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:31:14.164911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:31:14.165257 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:31:14.165441 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 20:31:14.166310 (+ 869us) server_negotiation.cc:299] Negotiation successful
0929 20:31:14.166588 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":2322,"threads_started":1}
I20260929 20:31:14.169070 8870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:31:14.141251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:38235 (local address 127.28.244.130:53163)
0929 20:31:14.142272 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0929 20:31:14.142330 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 20:31:14.144895 (+ 2565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:31:14.146314 (+ 1419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:31:14.146328 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:31:14.147163 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:31:14.148332 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:31:14.148372 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:31:14.150408 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:31:14.150416 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:31:14.152907 (+ 2491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:31:14.152930 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:31:14.154497 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:31:14.154508 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:31:14.154889 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:31:14.155871 (+ 982us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:31:14.155914 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:31:14.159342 (+ 3428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:31:14.163338 (+ 3996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:31:14.163350 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:31:14.163375 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:31:14.164053 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:31:14.164514 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:31:14.164520 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:31:14.164524 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:31:14.164742 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:31:14.165476 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:31:14.165486 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:31:14.165888 (+ 402us) client_negotiation.cc:769] Sending connection context
0929 20:31:14.166203 (+ 315us) client_negotiation.cc:240] Negotiation successful
0929 20:31:14.166846 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":422,"threads_started":1}
I20260929 20:31:14.170820 8867 heartbeater.cc:347] Connected to a master server at 127.28.244.190:38235
I20260929 20:31:14.171208 8867 heartbeater.cc:464] Registering TS with master...
I20260929 20:31:14.172144 8867 heartbeater.cc:511] Master 127.28.244.190:38235 requested a full tablet report, sending...
I20260929 20:31:14.173902 8511 ts_manager.cc:194] Registered new tserver with Master: a00ef07dc34d43ba8028808bd0a9d464 (127.28.244.130:34049)
WARNING: no policy specified for kudu/127.28.244.131@KRBTEST.COM; defaulting to no policy
I20260929 20:31:14.174875 8511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.130@KRBTEST.COM'} at 127.28.244.130:53163
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab.
Entry for principal kudu/127.28.244.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab.
I20260929 20:31:14.211372 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.244.148:56181/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.131:0
--local_ip_for_outbound_sockets=127.28.244.131
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:38235
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:43161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:31:14.411142 8724 heartbeater.cc:503] Master 127.28.244.190:38235 was elected leader, sending a full tablet report...
W20260929 20:31:14.614894 8873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:31:14.615219 8873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:31:14.615375 8873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:31:14.623613 8873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:31:14.623744 8873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:31:14.623947 8873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.131
I20260929 20:31:14.634814 8873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:43161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.244.148:56181/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:38235
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.131
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:31:14.636377 8873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:31:14.637814 8873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:31:14.655424 8878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:31:14.655424 8879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:31:14.655827 8881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:31:14.656270 8873 server_base.cc:1034] running on GCE node
I20260929 20:31:14.656948 8873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:31:14.658114 8873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:31:14.659370 8873 hybrid_clock.cc:648] HybridClock initialized: now 1790713874659303 us; error 69 us; skew 500 ppm
I20260929 20:31:14.659714 8873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:31:14 dist-test-slave-p29l krb5kdc[8035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713874, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:31:14.663352 8873 init.cc:377] Logged in from keytab as kudu/127.28.244.131@KRBTEST.COM (short username kudu)
I20260929 20:31:14.665155 8873 webserver.cc:460] Webserver started at http://127.28.244.131:43329/ using document root <none> and password file <none>
I20260929 20:31:14.665915 8873 fs_manager.cc:359] Metadata directory not provided
I20260929 20:31:14.666071 8873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:31:14.666399 8873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 20:31:14 dist-test-slave-p29l krb5kdc[8035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713874, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for HTTP/127.28.244.148@KRBTEST.COM
I20260929 20:31:14.687856 8873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-2/data/instance:
uuid: "3c76ed190e1b4a259d81178fb8f43584"
format_stamp: "Formatted at 2026-09-29 20:31:14 on dist-test-slave-p29l"
server_key: "982298500283e674ac1bc4a728f074ba"
server_key_iv: "db99f88520d31917997ec216341f2b8f"
server_key_version: "kuduclusterkey@0"
I20260929 20:31:14.688799 8873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance:
uuid: "3c76ed190e1b4a259d81178fb8f43584"
format_stamp: "Formatted at 2026-09-29 20:31:14 on dist-test-slave-p29l"
server_key: "982298500283e674ac1bc4a728f074ba"
server_key_iv: "db99f88520d31917997ec216341f2b8f"
server_key_version: "kuduclusterkey@0"
I20260929 20:31:14.695225 8873 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260929 20:31:14.719626 8888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:31:14.721791 8873 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 20:31:14.722018 8873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-2/wal
uuid: "3c76ed190e1b4a259d81178fb8f43584"
format_stamp: "Formatted at 2026-09-29 20:31:14 on dist-test-slave-p29l"
server_key: "982298500283e674ac1bc4a728f074ba"
server_key_iv: "db99f88520d31917997ec216341f2b8f"
server_key_version: "kuduclusterkey@0"
I20260929 20:31:14.722333 8873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:31:14.762832 8873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:31:14.769037 8873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:31:14.769539 8873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:31:14.770987 8873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:31:14.773195 8873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:31:14.773308 8873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:31:14.773440 8873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:31:14.773521 8873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:31:14.810920 8873 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.131:37423
I20260929 20:31:14.810940 9000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.131:37423 every 8 connection(s)
I20260929 20:31:14.812976 8873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
I20260929 20:31:14.820348 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 8873
I20260929 20:31:14.820650 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance
I20260929 20:31:14.821199 29650 external_mini_cluster.cc:1428] Setting key b208b27a28a9cc5e8631ee8d02da5e90
Sep 29 20:31:14 dist-test-slave-p29l krb5kdc[8035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713874, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:31:14.839668 9003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:31:14.815920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:46783 (local address 127.28.244.190:38235)
0929 20:31:14.816506 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0929 20:31:14.816513 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:31:14.817719 (+ 1206us) server_negotiation.cc:407] Connection header received
0929 20:31:14.820929 (+ 3210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:31:14.820935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:31:14.821049 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:31:14.821185 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:31:14.823414 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:31:14.824595 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:31:14.826245 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:31:14.826706 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:31:14.826842 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:31:14.831012 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:31:14.831021 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:31:14.831029 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:31:14.831084 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:31:14.835612 (+ 4528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:31:14.836719 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:31:14.836728 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:31:14.836734 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:31:14.836926 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:31:14.837460 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:31:14.837468 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:31:14.837481 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:31:14.837913 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:31:14.838088 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 20:31:14.838833 (+ 745us) server_negotiation.cc:299] Negotiation successful
0929 20:31:14.839125 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":283,"threads_started":1}
I20260929 20:31:14.840374 9004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:31:14.816505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:38235 (local address 127.28.244.131:46783)
0929 20:31:14.817495 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0929 20:31:14.817558 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 20:31:14.820551 (+ 2993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:31:14.821513 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:31:14.821531 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:31:14.822254 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:31:14.823219 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:31:14.823248 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:31:14.824772 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:31:14.824778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:31:14.826088 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:31:14.826102 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:31:14.826955 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:31:14.826965 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:31:14.827241 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:31:14.828104 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:31:14.828142 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:31:14.830744 (+ 2602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:31:14.835800 (+ 5056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:31:14.835842 (+ 42us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:31:14.835865 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:31:14.836516 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:31:14.837061 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:31:14.837069 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:31:14.837075 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:31:14.837325 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:31:14.838042 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:31:14.838049 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:31:14.838422 (+ 373us) client_negotiation.cc:769] Sending connection context
0929 20:31:14.838671 (+ 249us) client_negotiation.cc:240] Negotiation successful
0929 20:31:14.839100 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":355,"threads_started":1}
I20260929 20:31:14.841910 9001 heartbeater.cc:347] Connected to a master server at 127.28.244.190:38235
I20260929 20:31:14.842240 9001 heartbeater.cc:464] Registering TS with master...
I20260929 20:31:14.842989 9001 heartbeater.cc:511] Master 127.28.244.190:38235 requested a full tablet report, sending...
I20260929 20:31:14.844623 8511 ts_manager.cc:194] Registered new tserver with Master: 3c76ed190e1b4a259d81178fb8f43584 (127.28.244.131:37423)
I20260929 20:31:14.845547 8511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.131@KRBTEST.COM'} at 127.28.244.131:46783
I20260929 20:31:14.850101 29650 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:31:14.871500 9003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:31:14.855402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51782 (local address 127.28.244.190:38235)
0929 20:31:14.855638 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 20:31:14.855646 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:31:14.856166 (+ 520us) server_negotiation.cc:407] Connection header received
0929 20:31:14.856331 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:31:14.856340 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:31:14.856502 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:31:14.856630 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:31:14.858478 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:31:14.859956 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:31:14.861503 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:31:14.862103 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:31:14.862273 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:31:14.864985 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:31:14.864998 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:31:14.865002 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:31:14.865041 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:31:14.868322 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:31:14.869224 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:31:14.869230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:31:14.869234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:31:14.869389 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:31:14.869875 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:31:14.869881 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:31:14.869885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:31:14.870264 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:31:14.870540 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 20:31:14.870931 (+ 391us) server_negotiation.cc:299] Negotiation successful
0929 20:31:14.871173 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 20:31:14.876502 8511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:31:14.884452 8511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:31:14.913893 9015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:31:14.897971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34182 (local address 127.28.244.130:34049)
0929 20:31:14.898709 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0929 20:31:14.898719 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:31:14.899213 (+ 494us) server_negotiation.cc:407] Connection header received
0929 20:31:14.899584 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:31:14.899595 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:31:14.899964 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:31:14.900175 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:31:14.903060 (+ 2885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:31:14.906838 (+ 3778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:31:14.910956 (+ 4118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:31:14.912582 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:31:14.912758 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:31:14.912937 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 20:31:14.913341 (+ 404us) server_negotiation.cc:299] Negotiation successful
0929 20:31:14.913524 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":1052,"threads_started":1}
I20260929 20:31:14.913933 9014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:31:14.897742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:34049 (local address 127.0.0.1:34182)
0929 20:31:14.899055 (+ 1313us) negotiation.cc:107] Waiting for socket to connect
0929 20:31:14.899107 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 20:31:14.899349 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:31:14.902115 (+ 2766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:31:14.902128 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:31:14.902191 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:31:14.902776 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:31:14.902793 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:31:14.907107 (+ 4314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:31:14.907115 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:31:14.910763 (+ 3648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:31:14.910774 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:31:14.912804 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:31:14.912813 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:31:14.913076 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:31:14.913129 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 20:31:14.913281 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 20:31:14.913500 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1030,"thread_start_us":881,"threads_started":1}
I20260929 20:31:14.917640 9016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:31:14.898430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:37423 (local address 127.0.0.1:56070)
0929 20:31:14.899133 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0929 20:31:14.899151 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:31:14.899312 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:31:14.910252 (+ 10940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:31:14.910265 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:31:14.910304 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:31:14.910809 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:31:14.910822 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:31:14.912662 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:31:14.912670 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:31:14.914888 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:31:14.914902 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:31:14.916758 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:31:14.916766 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:31:14.916935 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:31:14.916971 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 20:31:14.917070 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 20:31:14.917271 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":1831,"threads_started":1}
I20260929 20:31:14.917696 9017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:31:14.898807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56070 (local address 127.28.244.131:37423)
0929 20:31:14.909163 (+ 10356us) server_negotiation.cc:206] Beginning negotiation
0929 20:31:14.909175 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:31:14.909225 (+ 50us) server_negotiation.cc:407] Connection header received
0929 20:31:14.909380 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:31:14.909393 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:31:14.909935 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:31:14.910894 (+ 959us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:31:14.911165 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:31:14.912528 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:31:14.915082 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:31:14.916521 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:31:14.916737 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:31:14.916913 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 20:31:14.917141 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 20:31:14.917315 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10144,"thread_start_us":9982,"threads_started":1}
I20260929 20:31:14.919178 9018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:31:14.899488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:36409 (local address 127.0.0.1:57580)
0929 20:31:14.910589 (+ 11101us) negotiation.cc:107] Waiting for socket to connect
0929 20:31:14.910620 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 20:31:14.910767 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:31:14.911476 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:31:14.911480 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:31:14.911514 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:31:14.911913 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:31:14.911925 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:31:14.913727 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:31:14.913733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:31:14.916013 (+ 2280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:31:14.916027 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:31:14.918444 (+ 2417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:31:14.918450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:31:14.918612 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:31:14.918647 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 20:31:14.918741 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 20:31:14.918885 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10945,"thread_start_us":10806,"threads_started":1}
I20260929 20:31:14.919363 9019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:31:14.900026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57580 (local address 127.28.244.129:36409)
0929 20:31:14.908709 (+ 8683us) server_negotiation.cc:206] Beginning negotiation
0929 20:31:14.908722 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:31:14.910753 (+ 2031us) server_negotiation.cc:407] Connection header received
0929 20:31:14.910993 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:31:14.911001 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:31:14.911251 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:31:14.911465 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:31:14.912092 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:31:14.913537 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:31:14.916203 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:31:14.917696 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:31:14.918122 (+ 426us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:31:14.918302 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 20:31:14.918861 (+ 559us) server_negotiation.cc:299] Negotiation successful
0929 20:31:14.919040 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8435,"thread_start_us":8259,"threads_started":1}
I20260929 20:31:14.918797 8802 tablet_service.cc:1467] Processing CreateTablet for tablet eb9ea6143d8a43a4a7f6da5cdacba636 (DEFAULT_TABLE table=test-table [id=5c7d44afb0ab42bf9ad6952ce572f4fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:31:14.921070 8802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb9ea6143d8a43a4a7f6da5cdacba636. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:31:14.921934 8936 tablet_service.cc:1467] Processing CreateTablet for tablet eb9ea6143d8a43a4a7f6da5cdacba636 (DEFAULT_TABLE table=test-table [id=5c7d44afb0ab42bf9ad6952ce572f4fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:31:14.923476 8936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb9ea6143d8a43a4a7f6da5cdacba636. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:31:14.923030 8659 tablet_service.cc:1467] Processing CreateTablet for tablet eb9ea6143d8a43a4a7f6da5cdacba636 (DEFAULT_TABLE table=test-table [id=5c7d44afb0ab42bf9ad6952ce572f4fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:31:14.924458 8659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb9ea6143d8a43a4a7f6da5cdacba636. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:31:14.934079 9020 tablet_bootstrap.cc:492] T eb9ea6143d8a43a4a7f6da5cdacba636 P a00ef07dc34d43ba8028808bd0a9d464: Bootstrap starting.
I20260929 20:31:14.938347 9020 tablet_bootstrap.cc:654] T eb9ea6143d8a43a4a7f6da5cdacba636 P a00ef07dc34d43ba8028808bd0a9d464: Neither blocks nor log segments found. Creating new log.
I20260929 20:31:14.939437 9021 tablet_bootstrap.cc:492] T eb9ea6143d8a43a4a7f6da5cdacba636 P 3c76ed190e1b4a259d81178fb8f43584: Bootstrap starting.
I20260929 20:31:14.939838 9020 log.cc:824] T eb9ea6143d8a43a4a7f6da5cdacba636 P a00ef07dc34d43ba8028808bd0a9d464: Log is configured to *not* fsync() on all Append() calls
I20260929 20:31:14.943276 9021 tablet_bootstrap.cc:654] T eb9ea6143d8a43a4a7f6da5cdacba636 P 3c76ed190e1b4a259d81178fb8f43584: Neither blocks nor log segments found. Creating new log.
I20260929 20:31:14.943781 9020 tablet_bootstrap.cc:492] T eb9ea6143d8a43a4a7f6da5cdacba636 P a00ef07dc34d43ba8028808bd0a9d464: No bootstrap required, opened a new log
I20260929 20:31:14.944121 9020 ts_tablet_manager.cc:1397] T eb9ea6143d8a43a4a7f6da5cdacba636 P a00ef07dc34d43ba8028808bd0a9d464: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.002s
I20260929 20:31:14.944397 9022 tablet_bootstrap.cc:492] T eb9ea6143d8a43a4a7f6da5cdacba636 P b92e4ee58b004430801ef2831b495dd9: Bootstrap starting.
I20260929 20:31:14.944631 9021 log.cc:824] T eb9ea6143d8a43a4a7f6da5cdacba636 P 3c76ed190e1b4a259d81178fb8f43584: Log is configured to *not* fsync() on all Append() calls
I20260929 20:31:14.947991 9022 tablet_bootstrap.cc:654] T eb9ea6143d8a43a4a7f6da5cdacba636 P b92e4ee58b004430801ef2831b495dd9: Neither blocks nor log segments found. Creating new log.
I20260929 20:31:14.948110 9021 tablet_bootstrap.cc:492] T eb9ea6143d8a43a4a7f6da5cdacba636 P 3c76ed190e1b4a259d81178fb8f43584: No bootstrap required, opened a new log
I20260929 20:31:14.948403 9021 ts_tablet_manager.cc:1397] T eb9ea6143d8a43a4a7f6da5cdacba636 P 3c76ed190e1b4a259d81178fb8f43584: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260929 20:31:14.949491 9022 log.cc:824] T eb9ea6143d8a43a4a7f6da5cdacba636 P b92e4ee58b004430801ef2831b495dd9: Log is configured to *not* fsync() on all Append() calls
I20260929 20:31:14.952554 9022 tablet_bootstrap.cc:492] T eb9ea6143d8a43a4a7f6da5cdacba636 P b92e4ee58b004430801ef2831b495dd9: No bootstrap required, opened a new log
I20260929 20:31:14.952843 9022 ts_tablet_manager.cc:1397] T eb9ea6143d8a43a4a7f6da5cdacba636 P b92e4ee58b004430801ef2831b495dd9: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 20:31:14.955482 9020 raft_consensus.cc:348] T eb9ea6143d8a43a4a7f6da5cdacba636 P a00ef07dc34d43ba8028808bd0a9d464 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00ef07dc34d43ba8028808bd0a9d464" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34049 } } peers { permanent_uuid: "b92e4ee58b004430801ef2831b495dd9" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 36409 } } peers { permanent_uuid: "3c76ed190e1b4a259d81178fb8f43584" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 37423 } }
I20260929 20:31:14.956022 9020 raft_consensus.cc:374] T eb9ea6143d8a43a4a7f6da5cdacba636 P a00ef07dc34d43ba8028808bd0a9d464 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:31:14.956130 9020 raft_consensus.cc:729] T eb9ea6143d8a43a4a7f6da5cdacba636 P a00ef07dc34d43ba8028808bd0a9d464 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a00ef07dc34d43ba8028808bd0a9d464, State: Initialized, Role: FOLLOWER
I20260929 20:31:14.956811 9020 consensus_queue.cc:260] T eb9ea6143d8a43a4a7f6da5cdacba636 P a00ef07dc34d43ba8028808bd0a9d464 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00ef07dc34d43ba8028808bd0a9d464" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34049 } } peers { permanent_uuid: "b92e4ee58b004430801ef2831b495dd9" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 36409 } } peers { permanent_uuid: "3c76ed190e1b4a259d81178fb8f43584" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 37423 } }
I20260929 20:31:14.958032 8867 heartbeater.cc:503] Master 127.28.244.190:38235 was elected leader, sending a full tablet report...
I20260929 20:31:14.958724 9020 ts_tablet_manager.cc:1428] T eb9ea6143d8a43a4a7f6da5cdacba636 P a00ef07dc34d43ba8028808bd0a9d464: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260929 20:31:14.959304 9021 raft_consensus.cc:348] T eb9ea6143d8a43a4a7f6da5cdacba636 P 3c76ed190e1b4a259d81178fb8f43584 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00ef07dc34d43ba8028808bd0a9d464" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34049 } } peers { permanent_uuid: "b92e4ee58b004430801ef2831b495dd9" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 36409 } } peers { permanent_uuid: "3c76ed190e1b4a259d81178fb8f43584" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 37423 } }
I20260929 20:31:14.959794 9021 raft_consensus.cc:374] T eb9ea6143d8a43a4a7f6da5cdacba636 P 3c76ed190e1b4a259d81178fb8f43584 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:31:14.959949 9021 raft_consensus.cc:729] T eb9ea6143d8a43a4a7f6da5cdacba636 P 3c76ed190e1b4a259d81178fb8f43584 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c76ed190e1b4a259d81178fb8f43584, State: Initialized, Role: FOLLOWER
I20260929 20:31:14.960620 9021 consensus_queue.cc:260] T eb9ea6143d8a43a4a7f6da5cdacba636 P 3c76ed190e1b4a259d81178fb8f43584 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00ef07dc34d43ba8028808bd0a9d464" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34049 } } peers { permanent_uuid: "b92e4ee58b004430801ef2831b495dd9" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 36409 } } peers { permanent_uuid: "3c76ed190e1b4a259d81178fb8f43584" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 37423 } }
I20260929 20:31:14.961791 9022 raft_consensus.cc:348] T eb9ea6143d8a43a4a7f6da5cdacba636 P b92e4ee58b004430801ef2831b495dd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00ef07dc34d43ba8028808bd0a9d464" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34049 } } peers { permanent_uuid: "b92e4ee58b004430801ef2831b495dd9" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 36409 } } peers { permanent_uuid: "3c76ed190e1b4a259d81178fb8f43584" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 37423 } }
I20260929 20:31:14.962303 9022 raft_consensus.cc:374] T eb9ea6143d8a43a4a7f6da5cdacba636 P b92e4ee58b004430801ef2831b495dd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:31:14.962409 9022 raft_consensus.cc:729] T eb9ea6143d8a43a4a7f6da5cdacba636 P b92e4ee58b004430801ef2831b495dd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92e4ee58b004430801ef2831b495dd9, State: Initialized, Role: FOLLOWER
I20260929 20:31:14.963073 9022 consensus_queue.cc:260] T eb9ea6143d8a43a4a7f6da5cdacba636 P b92e4ee58b004430801ef2831b495dd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00ef07dc34d43ba8028808bd0a9d464" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34049 } } peers { permanent_uuid: "b92e4ee58b004430801ef2831b495dd9" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 36409 } } peers { permanent_uuid: "3c76ed190e1b4a259d81178fb8f43584" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 37423 } }
I20260929 20:31:14.963845 9001 heartbeater.cc:503] Master 127.28.244.190:38235 was elected leader, sending a full tablet report...
I20260929 20:31:14.964584 9021 ts_tablet_manager.cc:1428] T eb9ea6143d8a43a4a7f6da5cdacba636 P 3c76ed190e1b4a259d81178fb8f43584: Time spent starting tablet: real 0.016s user 0.013s sys 0.003s
I20260929 20:31:14.966336 9022 ts_tablet_manager.cc:1428] T eb9ea6143d8a43a4a7f6da5cdacba636 P b92e4ee58b004430801ef2831b495dd9: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260929 20:31:14.977456 9028 raft_consensus.cc:482] T eb9ea6143d8a43a4a7f6da5cdacba636 P b92e4ee58b004430801ef2831b495dd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:31:14.977882 9028 raft_consensus.cc:504] T eb9ea6143d8a43a4a7f6da5cdacba636 P b92e4ee58b004430801ef2831b495dd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00ef07dc34d43ba8028808bd0a9d464" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34049 } } peers { permanent_uuid: "b92e4ee58b004430801ef2831b495dd9" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 36409 } } peers { permanent_uuid: "3c76ed190e1b4a259d81178fb8f43584" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 37423 } }
I20260929 20:31:14.979971 9028 leader_election.cc:288] T eb9ea6143d8a43a4a7f6da5cdacba636 P b92e4ee58b004430801ef2831b495dd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a00ef07dc34d43ba8028808bd0a9d464 (127.28.244.130:34049), 3c76ed190e1b4a259d81178fb8f43584 (127.28.244.131:37423)
I20260929 20:31:14.988880 9017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:31:14.980910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:39179 (local address 127.28.244.131:37423)
0929 20:31:14.981134 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 20:31:14.981142 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:31:14.981712 (+ 570us) server_negotiation.cc:407] Connection header received
0929 20:31:14.981896 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:31:14.981901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:31:14.981985 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:31:14.982088 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:31:14.982682 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:31:14.983683 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:31:14.986391 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:31:14.987646 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:31:14.987838 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:31:14.987918 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 20:31:14.988483 (+ 565us) server_negotiation.cc:299] Negotiation successful
0929 20:31:14.988615 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 20:31:14.988885 9015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:31:14.980406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:42613 (local address 127.28.244.130:34049)
0929 20:31:14.980643 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 20:31:14.980652 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:31:14.981224 (+ 572us) server_negotiation.cc:407] Connection header received
0929 20:31:14.981593 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:31:14.981603 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:31:14.981741 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:31:14.981853 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:31:14.983210 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:31:14.984026 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:31:14.986392 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:31:14.987646 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:31:14.987838 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:31:14.987918 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 20:31:14.988483 (+ 565us) server_negotiation.cc:299] Negotiation successful
0929 20:31:14.988635 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:31:14.989152 9030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:31:14.980459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:37423 (local address 127.28.244.129:39179)
0929 20:31:14.981609 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0929 20:31:14.981633 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:31:14.981763 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:31:14.982114 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:31:14.982118 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:31:14.982145 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:31:14.982496 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:31:14.982509 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:31:14.983885 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:31:14.983890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:31:14.986217 (+ 2327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:31:14.986233 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:31:14.987967 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:31:14.987974 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:31:14.988248 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:31:14.988344 (+ 96us) client_negotiation.cc:769] Sending connection context
0929 20:31:14.988444 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 20:31:14.988648 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1021,"mutex_wait_us":99,"thread_start_us":314,"threads_started":1}
I20260929 20:31:14.989176 9029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:31:14.980406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:34049 (local address 127.28.244.129:42613)
0929 20:31:14.981105 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0929 20:31:14.981126 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:31:14.981367 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:31:14.982604 (+ 1237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:31:14.982611 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:31:14.982642 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:31:14.983038 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:31:14.983052 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:31:14.984150 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:31:14.984157 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:31:14.986241 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:31:14.986250 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:31:14.987978 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:31:14.987983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:31:14.988250 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:31:14.988338 (+ 88us) client_negotiation.cc:769] Sending connection context
0929 20:31:14.988444 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 20:31:14.988648 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":118,"thread_start_us":875,"threads_started":1}
I20260929 20:31:14.990239 8822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb9ea6143d8a43a4a7f6da5cdacba636" candidate_uuid: "b92e4ee58b004430801ef2831b495dd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a00ef07dc34d43ba8028808bd0a9d464" is_pre_election: true
I20260929 20:31:14.990466 8956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb9ea6143d8a43a4a7f6da5cdacba636" candidate_uuid: "b92e4ee58b004430801ef2831b495dd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c76ed190e1b4a259d81178fb8f43584" is_pre_election: true
I20260929 20:31:14.990803 8822 raft_consensus.cc:2454] T eb9ea6143d8a43a4a7f6da5cdacba636 P a00ef07dc34d43ba8028808bd0a9d464 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b92e4ee58b004430801ef2831b495dd9 in term 0.
I20260929 20:31:14.991001 8956 raft_consensus.cc:2454] T eb9ea6143d8a43a4a7f6da5cdacba636 P 3c76ed190e1b4a259d81178fb8f43584 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b92e4ee58b004430801ef2831b495dd9 in term 0.
I20260929 20:31:14.991523 8615 leader_election.cc:302] T eb9ea6143d8a43a4a7f6da5cdacba636 P b92e4ee58b004430801ef2831b495dd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a00ef07dc34d43ba8028808bd0a9d464, b92e4ee58b004430801ef2831b495dd9; no voters:
I20260929 20:31:14.991971 9028 raft_consensus.cc:2781] T eb9ea6143d8a43a4a7f6da5cdacba636 P b92e4ee58b004430801ef2831b495dd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:31:14.992131 9028 raft_consensus.cc:482] T eb9ea6143d8a43a4a7f6da5cdacba636 P b92e4ee58b004430801ef2831b495dd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:31:14.992240 9028 raft_consensus.cc:3037] T eb9ea6143d8a43a4a7f6da5cdacba636 P b92e4ee58b004430801ef2831b495dd9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:31:14.994432 9028 raft_consensus.cc:504] T eb9ea6143d8a43a4a7f6da5cdacba636 P b92e4ee58b004430801ef2831b495dd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00ef07dc34d43ba8028808bd0a9d464" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34049 } } peers { permanent_uuid: "b92e4ee58b004430801ef2831b495dd9" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 36409 } } peers { permanent_uuid: "3c76ed190e1b4a259d81178fb8f43584" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 37423 } }
I20260929 20:31:14.995138 9028 leader_election.cc:288] T eb9ea6143d8a43a4a7f6da5cdacba636 P b92e4ee58b004430801ef2831b495dd9 [CANDIDATE]: Term 1 election: Requested vote from peers a00ef07dc34d43ba8028808bd0a9d464 (127.28.244.130:34049), 3c76ed190e1b4a259d81178fb8f43584 (127.28.244.131:37423)
I20260929 20:31:14.995603 8822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb9ea6143d8a43a4a7f6da5cdacba636" candidate_uuid: "b92e4ee58b004430801ef2831b495dd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a00ef07dc34d43ba8028808bd0a9d464"
I20260929 20:31:14.995680 8956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb9ea6143d8a43a4a7f6da5cdacba636" candidate_uuid: "b92e4ee58b004430801ef2831b495dd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c76ed190e1b4a259d81178fb8f43584"
I20260929 20:31:14.995812 8822 raft_consensus.cc:3037] T eb9ea6143d8a43a4a7f6da5cdacba636 P a00ef07dc34d43ba8028808bd0a9d464 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:31:14.995947 8956 raft_consensus.cc:3037] T eb9ea6143d8a43a4a7f6da5cdacba636 P 3c76ed190e1b4a259d81178fb8f43584 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:31:14.998193 8956 raft_consensus.cc:2454] T eb9ea6143d8a43a4a7f6da5cdacba636 P 3c76ed190e1b4a259d81178fb8f43584 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b92e4ee58b004430801ef2831b495dd9 in term 1.
I20260929 20:31:14.998198 8822 raft_consensus.cc:2454] T eb9ea6143d8a43a4a7f6da5cdacba636 P a00ef07dc34d43ba8028808bd0a9d464 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b92e4ee58b004430801ef2831b495dd9 in term 1.
I20260929 20:31:14.998669 8614 leader_election.cc:302] T eb9ea6143d8a43a4a7f6da5cdacba636 P b92e4ee58b004430801ef2831b495dd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c76ed190e1b4a259d81178fb8f43584, b92e4ee58b004430801ef2831b495dd9; no voters:
I20260929 20:31:14.999013 9028 raft_consensus.cc:2781] T eb9ea6143d8a43a4a7f6da5cdacba636 P b92e4ee58b004430801ef2831b495dd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:31:14.999548 9028 raft_consensus.cc:686] T eb9ea6143d8a43a4a7f6da5cdacba636 P b92e4ee58b004430801ef2831b495dd9 [term 1 LEADER]: Becoming Leader. State: Replica: b92e4ee58b004430801ef2831b495dd9, State: Running, Role: LEADER
I20260929 20:31:15.000218 9028 consensus_queue.cc:237] T eb9ea6143d8a43a4a7f6da5cdacba636 P b92e4ee58b004430801ef2831b495dd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00ef07dc34d43ba8028808bd0a9d464" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34049 } } peers { permanent_uuid: "b92e4ee58b004430801ef2831b495dd9" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 36409 } } peers { permanent_uuid: "3c76ed190e1b4a259d81178fb8f43584" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 37423 } }
I20260929 20:31:15.005172 8511 catalog_manager.cc:5495] T eb9ea6143d8a43a4a7f6da5cdacba636 P b92e4ee58b004430801ef2831b495dd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b92e4ee58b004430801ef2831b495dd9 (127.28.244.129). New cstate: current_term: 1 leader_uuid: "b92e4ee58b004430801ef2831b495dd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00ef07dc34d43ba8028808bd0a9d464" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b92e4ee58b004430801ef2831b495dd9" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 36409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c76ed190e1b4a259d81178fb8f43584" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 37423 } health_report { overall_health: UNKNOWN } } }
I20260929 20:31:15.041154 9019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:31:15.035051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57592 (local address 127.28.244.129:36409)
0929 20:31:15.035285 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 20:31:15.035292 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:31:15.035348 (+ 56us) server_negotiation.cc:407] Connection header received
0929 20:31:15.035657 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:31:15.035662 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:31:15.035749 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:31:15.035935 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:31:15.036365 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:31:15.037109 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:31:15.038623 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:31:15.039041 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:31:15.039195 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:31:15.039860 (+ 665us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:31:15.040468 (+ 608us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:31:15.040634 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 20:31:15.040772 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 20:31:15.040862 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 20:31:15.050489 8822 raft_consensus.cc:1261] T eb9ea6143d8a43a4a7f6da5cdacba636 P a00ef07dc34d43ba8028808bd0a9d464 [term 1 FOLLOWER]: Refusing update from remote peer b92e4ee58b004430801ef2831b495dd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:31:15.050489 8956 raft_consensus.cc:1261] T eb9ea6143d8a43a4a7f6da5cdacba636 P 3c76ed190e1b4a259d81178fb8f43584 [term 1 FOLLOWER]: Refusing update from remote peer b92e4ee58b004430801ef2831b495dd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:31:15.051599 9028 consensus_queue.cc:1035] T eb9ea6143d8a43a4a7f6da5cdacba636 P b92e4ee58b004430801ef2831b495dd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c76ed190e1b4a259d81178fb8f43584" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 37423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:31:15.051934 9031 consensus_queue.cc:1035] T eb9ea6143d8a43a4a7f6da5cdacba636 P b92e4ee58b004430801ef2831b495dd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a00ef07dc34d43ba8028808bd0a9d464" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 34049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 20:31:15.064771 9002 tablet.cc:2369] T eb9ea6143d8a43a4a7f6da5cdacba636 P 3c76ed190e1b4a259d81178fb8f43584: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:31:15.066831 9034 mvcc.cc:205] Tried to move back new op lower bound from 7334764032194269184 to 7334764032004694016. Current Snapshot: MvccSnapshot[applied={T|T < 7334764032194269184}]
I20260929 20:31:15.077021 9035 mvcc.cc:205] Tried to move back new op lower bound from 7334764032194269184 to 7334764032004694016. Current Snapshot: MvccSnapshot[applied={T|T < 7334764032194269184}]
I20260929 20:31:15.090353 8511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:31:15.090593 8511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:31:15.094489 8511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7229fd9a18046b98cee658a29ca7936: Sending DeleteTablet for 3 replicas of tablet eb9ea6143d8a43a4a7f6da5cdacba636
I20260929 20:31:15.095899 8802 tablet_service.cc:1514] Processing DeleteTablet for tablet eb9ea6143d8a43a4a7f6da5cdacba636 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:31:15 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:34182
I20260929 20:31:15.095984 8659 tablet_service.cc:1514] Processing DeleteTablet for tablet eb9ea6143d8a43a4a7f6da5cdacba636 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:31:15 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:57580
I20260929 20:31:15.096110 8936 tablet_service.cc:1514] Processing DeleteTablet for tablet eb9ea6143d8a43a4a7f6da5cdacba636 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:31:15 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:56070
I20260929 20:31:15.098667 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 8591
I20260929 20:31:15.108610 9046 tablet_replica.cc:330] T eb9ea6143d8a43a4a7f6da5cdacba636 P 3c76ed190e1b4a259d81178fb8f43584: stopping tablet replica
I20260929 20:31:15.108773 9045 tablet_replica.cc:330] T eb9ea6143d8a43a4a7f6da5cdacba636 P b92e4ee58b004430801ef2831b495dd9: stopping tablet replica
I20260929 20:31:15.109283 9046 raft_consensus.cc:2229] T eb9ea6143d8a43a4a7f6da5cdacba636 P 3c76ed190e1b4a259d81178fb8f43584 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:31:15.109292 9045 raft_consensus.cc:2229] T eb9ea6143d8a43a4a7f6da5cdacba636 P b92e4ee58b004430801ef2831b495dd9 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:31:15.109891 9046 raft_consensus.cc:2258] T eb9ea6143d8a43a4a7f6da5cdacba636 P 3c76ed190e1b4a259d81178fb8f43584 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:31:15.110039 9045 raft_consensus.cc:2258] T eb9ea6143d8a43a4a7f6da5cdacba636 P b92e4ee58b004430801ef2831b495dd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:31:15.112250 9046 ts_tablet_manager.cc:1905] T eb9ea6143d8a43a4a7f6da5cdacba636 P 3c76ed190e1b4a259d81178fb8f43584: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:31:15.112258 9045 ts_tablet_manager.cc:1905] T eb9ea6143d8a43a4a7f6da5cdacba636 P b92e4ee58b004430801ef2831b495dd9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:31:15.119997 9046 ts_tablet_manager.cc:1918] T eb9ea6143d8a43a4a7f6da5cdacba636 P 3c76ed190e1b4a259d81178fb8f43584: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:31:15.119997 9045 ts_tablet_manager.cc:1918] T eb9ea6143d8a43a4a7f6da5cdacba636 P b92e4ee58b004430801ef2831b495dd9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:31:15.120215 9046 log.cc:1196] T eb9ea6143d8a43a4a7f6da5cdacba636 P 3c76ed190e1b4a259d81178fb8f43584: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-2/wal/wals/eb9ea6143d8a43a4a7f6da5cdacba636
I20260929 20:31:15.120226 9045 log.cc:1196] T eb9ea6143d8a43a4a7f6da5cdacba636 P b92e4ee58b004430801ef2831b495dd9: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-0/wal/wals/eb9ea6143d8a43a4a7f6da5cdacba636
I20260929 20:31:15.120795 9046 ts_tablet_manager.cc:1939] T eb9ea6143d8a43a4a7f6da5cdacba636 P 3c76ed190e1b4a259d81178fb8f43584: Deleting consensus metadata
I20260929 20:31:15.120802 9045 ts_tablet_manager.cc:1939] T eb9ea6143d8a43a4a7f6da5cdacba636 P b92e4ee58b004430801ef2831b495dd9: Deleting consensus metadata
I20260929 20:31:15.122784 8498 catalog_manager.cc:4841] TS b92e4ee58b004430801ef2831b495dd9 (127.28.244.129:36409): tablet eb9ea6143d8a43a4a7f6da5cdacba636 (table test-table [id=5c7d44afb0ab42bf9ad6952ce572f4fa]) successfully deleted
I20260929 20:31:15.122917 8497 catalog_manager.cc:4841] TS 3c76ed190e1b4a259d81178fb8f43584 (127.28.244.131:37423): tablet eb9ea6143d8a43a4a7f6da5cdacba636 (table test-table [id=5c7d44afb0ab42bf9ad6952ce572f4fa]) successfully deleted
I20260929 20:31:15.123725 9044 tablet_replica.cc:330] T eb9ea6143d8a43a4a7f6da5cdacba636 P a00ef07dc34d43ba8028808bd0a9d464: stopping tablet replica
I20260929 20:31:15.124245 9044 raft_consensus.cc:2229] T eb9ea6143d8a43a4a7f6da5cdacba636 P a00ef07dc34d43ba8028808bd0a9d464 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:31:15.124924 9044 raft_consensus.cc:2258] T eb9ea6143d8a43a4a7f6da5cdacba636 P a00ef07dc34d43ba8028808bd0a9d464 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:31:15.127207 9044 ts_tablet_manager.cc:1905] T eb9ea6143d8a43a4a7f6da5cdacba636 P a00ef07dc34d43ba8028808bd0a9d464: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:31:15.134419 9044 ts_tablet_manager.cc:1918] T eb9ea6143d8a43a4a7f6da5cdacba636 P a00ef07dc34d43ba8028808bd0a9d464: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:31:15.134641 9044 log.cc:1196] T eb9ea6143d8a43a4a7f6da5cdacba636 P a00ef07dc34d43ba8028808bd0a9d464: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713721201454-29650-0/minicluster-data/ts-1/wal/wals/eb9ea6143d8a43a4a7f6da5cdacba636
I20260929 20:31:15.135267 9044 ts_tablet_manager.cc:1939] T eb9ea6143d8a43a4a7f6da5cdacba636 P a00ef07dc34d43ba8028808bd0a9d464: Deleting consensus metadata
Sep 29 20:31:15 dist-test-slave-p29l krb5kdc[8035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.129@KRBTEST.COM
I20260929 20:31:15.137905 8498 catalog_manager.cc:4841] TS a00ef07dc34d43ba8028808bd0a9d464 (127.28.244.130:34049): tablet eb9ea6143d8a43a4a7f6da5cdacba636 (table test-table [id=5c7d44afb0ab42bf9ad6952ce572f4fa]) successfully deleted
I20260929 20:31:15.144533 9019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:31:15.102274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57598 (local address 127.28.244.129:36409)
0929 20:31:15.103373 (+ 1099us) server_negotiation.cc:206] Beginning negotiation
0929 20:31:15.103381 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:31:15.123781 (+ 20400us) server_negotiation.cc:407] Connection header received
0929 20:31:15.124090 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:31:15.124099 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:31:15.124199 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:31:15.124324 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:31:15.126120 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:31:15.128662 (+ 2542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:31:15.130687 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:31:15.131402 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:31:15.131574 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:31:15.137404 (+ 5830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:31:15.137430 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:31:15.137453 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:31:15.137514 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:31:15.141225 (+ 3711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:31:15.142073 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:31:15.142082 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:31:15.142084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:31:15.142255 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:31:15.142712 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:31:15.142719 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:31:15.142727 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:31:15.143145 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:31:15.143481 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0929 20:31:15.143872 (+ 391us) server_negotiation.cc:299] Negotiation successful
0929 20:31:15.144147 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":950}
I20260929 20:31:15.145375 8719 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:31:15.248422 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 8591
I20260929 20:31:15.268469 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 8730
Sep 29 20:31:15 dist-test-slave-p29l krb5kdc[8035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.130@KRBTEST.COM
I20260929 20:31:15.285405 9015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:31:15.269526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34184 (local address 127.28.244.130:34049)
0929 20:31:15.269751 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 20:31:15.269758 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:31:15.269910 (+ 152us) server_negotiation.cc:407] Connection header received
0929 20:31:15.270132 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:31:15.270137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:31:15.270217 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:31:15.270325 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:31:15.271568 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:31:15.272455 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:31:15.273782 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:31:15.274186 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:31:15.274325 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:31:15.278906 (+ 4581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:31:15.278923 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:31:15.278955 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:31:15.279011 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:31:15.282249 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:31:15.283119 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:31:15.283127 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:31:15.283130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:31:15.283285 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:31:15.283731 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:31:15.283736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:31:15.283739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:31:15.284210 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:31:15.284450 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 20:31:15.284809 (+ 359us) server_negotiation.cc:299] Negotiation successful
0929 20:31:15.284997 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 20:31:15.286103 8862 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:31:15.379706 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 8730
I20260929 20:31:15.397464 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 8873
Sep 29 20:31:15 dist-test-slave-p29l krb5kdc[8035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.131@KRBTEST.COM
I20260929 20:31:15.414460 9017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:31:15.398321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56072 (local address 127.28.244.131:37423)
0929 20:31:15.398513 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 20:31:15.398566 (+ 53us) server_negotiation.cc:399] Waiting for connection header
0929 20:31:15.398586 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:31:15.398786 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:31:15.398791 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:31:15.398866 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:31:15.398973 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:31:15.400609 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:31:15.401375 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:31:15.402741 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:31:15.403215 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:31:15.403350 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:31:15.408237 (+ 4887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:31:15.408257 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:31:15.408275 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:31:15.408324 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:31:15.411404 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:31:15.412343 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:31:15.412352 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:31:15.412354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:31:15.412516 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:31:15.412898 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:31:15.412901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:31:15.412903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:31:15.413327 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:31:15.413557 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 20:31:15.413956 (+ 399us) server_negotiation.cc:299] Negotiation successful
0929 20:31:15.414147 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:31:15.415227 8996 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:31:15.506024 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 8873
I20260929 20:31:15.523442 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 8479
I20260929 20:31:15.525017 8541 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:31:15.642354 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 8479
I20260929 20:31:15.667533 29650 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 20:31:16.004094 29650 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 20:31:16.004318 29650 mini_ranger.cc:66] Stopping Ranger...
I20260929 20:31:16.351001 29650 mini_ranger.cc:68] Stopped Ranger
2026-09-29 20:31:16.351 UTC [8065] LOG: received smart shutdown request
2026-09-29 20:31:16.355 UTC [8065] LOG: background worker "logical replication launcher" (PID 8074) exited with exit code 1
2026-09-29 20:31:16.355 UTC [8069] LOG: shutting down
2026-09-29 20:31:16.549 UTC [8065] LOG: database system is shut down
2026-09-29T20:31:16Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (51839 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:31:16 dist-test-slave-p29l krb5kdc[9060](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:31:16 dist-test-slave-p29l krb5kdc[9060](info): set up 2 sockets
Sep 29 20:31:16 dist-test-slave-p29l krb5kdc[9060](info): commencing operation
krb5kdc: starting...
W20260929 20:31:17.824220 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:31:17 dist-test-slave-p29l krb5kdc[9060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:31:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:31:18Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.244.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.244.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.244.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/rangeradmin_127.28.244.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.244.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/rangeradmin_127.28.244.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.244.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.244.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.244.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/rangerlookup_127.28.244.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.244.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/rangerlookup_127.28.244.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.244.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.244.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.244.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/HTTP_127.28.244.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.244.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/HTTP_127.28.244.148@KRBTEST.COM.keytab.
I20260929 20:31:18.184937 29650 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskNrwVME/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/postgres -l logfile start
2026-09-29 20:31:21.013 UTC [9090] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 20:31:21.013 UTC [9090] LOG: listening on IPv4 address "127.28.244.148", port 32861
2026-09-29 20:31:21.017 UTC [9090] LOG: listening on Unix socket "/tmp/.s.PGSQL.32861"
2026-09-29 20:31:21.041 UTC [9093] LOG: database system was shut down at 2026-09-29 20:31:19 UTC
2026-09-29 20:31:21.045 UTC [9090] LOG: database system is ready to accept connections
I20260929 20:31:22.043608 29650 mini_postgres.cc:96] Postgres bound to 32861
2026-09-29 20:31:22.061 UTC [9101] FATAL: database "slave" does not exist
127.28.244.148:32861 - accepting connections
I20260929 20:31:22.062561 29650 mini_ranger.cc:161] Starting Ranger...
I20260929 20:31:22.092986 29650 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 20:31:22.685226 29650 mini_ranger.cc:87] Created ranger Postgres database
I20260929 20:31:22.685380 29650 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ranger-admin
2026-09-29 20:31:23,003 [I] DB FLAVOR :POSTGRES
2026-09-29 20:31:23,004 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 20:31:23,004 [I] Checking connection..
2026-09-29 20:31:23,004 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:32861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 20:31:23,282 [I] Checking connection passed.
2026-09-29 20:31:23,282 [I] --------- Verifying version history table ---------
2026-09-29 20:31:23,282 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:32861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 20:31:23,572 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 20:31:23,573 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:32861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 20:31:23,853 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 20:31:23,854 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 20:31:23,854 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:32861/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 20:31:24.117 UTC [9165] WARNING: there is no transaction in progress
2026-09-29 20:31:24,124 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 20:31:24,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:32861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 20:31:24,383 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 20:31:24,384 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 20:31:24,384 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:32861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 20:31:24,671 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:32861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 20:31:24,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:32861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-p29l.c.gcp-upstream.internal','N') ;"
2026-09-29 20:31:25,207 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:32861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 20:31:25,459 [I] Table x_portal_user does not exist in database ranger
2026-09-29 20:31:25,459 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:32861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 20:31:25,718 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 20:31:25,718 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:32861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 20:31:25,976 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:32861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 20:31:26,240 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 20:31:26,240 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:32861/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 20:31:27.278 UTC [9273] WARNING: there is no transaction in progress
2026-09-29 20:31:27.295 UTC [9273] WARNING: there is no transaction in progress
2026-09-29 20:31:27.314 UTC [9273] WARNING: there is no transaction in progress
2026-09-29 20:31:27.330 UTC [9273] WARNING: there is no transaction in progress
2026-09-29 20:31:27.350 UTC [9273] WARNING: there is no transaction in progress
2026-09-29 20:31:27.371 UTC [9273] WARNING: there is no transaction in progress
2026-09-29 20:31:27.430 UTC [9273] WARNING: there is no transaction in progress
2026-09-29 20:31:27.438 UTC [9273] WARNING: there is no transaction in progress
2026-09-29 20:31:27.455 UTC [9273] WARNING: there is no transaction in progress
2026-09-29 20:31:27.466 UTC [9273] WARNING: there is no transaction in progress
2026-09-29 20:31:27.478 UTC [9273] WARNING: there is no transaction in progress
2026-09-29 20:31:27.493 UTC [9273] WARNING: there is no transaction in progress
2026-09-29 20:31:27.503 UTC [9273] WARNING: there is no transaction in progress
2026-09-29 20:31:27.510 UTC [9273] WARNING: there is no transaction in progress
2026-09-29 20:31:27.517 UTC [9273] WARNING: there is no transaction in progress
2026-09-29 20:31:27.525 UTC [9273] WARNING: there is no transaction in progress
2026-09-29 20:31:28,263 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 20:31:28,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:32861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 20:31:28,527 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 20:31:28,528 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:32861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 20:31:28,804 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 20:31:28,804 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:32861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 20:31:29,056 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 20:31:29,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:32861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 20:31:29,318 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:32861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 20:31:29,591 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:32861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-p29l.c.gcp-upstream.internal';"
2026-09-29 20:31:29,842 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 20:31:29,843 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:32861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 20:31:30,104 [I] DB_PATCHES have already been applied
I20260929 20:31:30.110690 29650 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ranger-admin:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskNrwVME/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 20:31:30.110788 29650 mini_ranger.cc:193] Using host: 127.28.244.148
I20260929 20:31:30.125978 29650 mini_ranger.cc:227] Ranger admin URL: http://127.28.244.148:43429
Sep 29, 2026 8:31:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 8:31:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 8:31:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 8:31:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 8:31:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 8:31:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 8:31:31 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 20:31:31.294315 29650 mini_ranger.cc:160] Time spent starting Ranger: real 9.232s user 0.000s sys 0.043s
Sep 29, 2026 8:31:52 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 8:31:52 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 8:31:52 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 8:31:52 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 8:31:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 8:31:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 8:31:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 8:31:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 8:31:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 8:31:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 8:31:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 8:31:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 8:31:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 8:31:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 8:31:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 8:31:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 8:31:53 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 8:31:54 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 20:31:54.671648 29650 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.244.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.244.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.244.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/rangerkms_127.28.244.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.244.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/rangerkms_127.28.244.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.244.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.244.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.244.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/HTTP_127.28.244.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.244.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/HTTP_127.28.244.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 20:31:54 dist-test-slave-p29l krb5kdc[9060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713914, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 20:31:54.858078 29650 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 20:31:54.879477 29650 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 20:31:55.791177 29650 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 20:31:55.791317 29650 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ranger-kms
I20260929 20:31:55.791370 29650 mini_ranger_kms.cc:227] Using postgres at 127.28.244.148:32861
2026-09-29 20:31:56,174 [I] DB FLAVOR :POSTGRES
2026-09-29 20:31:56,174 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 20:31:56,174 [I] Checking connection
2026-09-29 20:31:56,174 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:32861/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 20:31:56,441 [I] connection success
2026-09-29 20:31:56,441 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 20:31:56,442 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 20:31:56,442 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:32861/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 20:31:56,679 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 20:31:56,679 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 20:31:56,679 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 20:31:56,679 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.244.148:32861/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 20:31:56,967 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 20:31:57.511314 29650 mini_ranger_kms.cc:310] Created kms service
I20260929 20:31:57.756750 29650 mini_ranger_kms.cc:326] Created kudu user
I20260929 20:31:57.806895 29650 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 20:31:57 dist-test-slave-p29l krb5kdc[9060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713914, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.244.148@KRBTEST.COM
I20260929 20:31:58.236047 29650 mini_ranger_kms.cc:384] Added ranger policy
I20260929 20:31:58.236407 29650 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ranger-kms:/tmp/dist-test-taskNrwVME/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskNrwVME/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 20:31:58.236459 29650 mini_ranger_kms.cc:240] Using host: 127.28.244.148
I20260929 20:31:58.255338 29650 mini_ranger_kms.cc:276] Ranger KMS PID: 9454
I20260929 20:31:58.255486 29650 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.244.148:34755
Sep 29, 2026 8:31:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 8:31:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 8:31:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 8:31:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 8:31:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 8:31:59 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 20:32:00.648609 29650 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.791s user 0.067s sys 0.325s
I20260929 20:32:00.648881 29650 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 20:32:00.649056 29650 mini_ranger_kms.cc:401] 127.28.244.148:34755/kms/v1/keys
Sep 29 20:32:00 dist-test-slave-p29l krb5kdc[9060](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790713920, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.244.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 20:32:01 dist-test-slave-p29l krb5kdc[9060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713920, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.244.148@KRBTEST.COM for HTTP/127.28.244.148@KRBTEST.COM
Sep 29 20:32:01 dist-test-slave-p29l krb5kdc[9060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713920, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.244.148@KRBTEST.COM for HTTP/127.28.244.148@KRBTEST.COM
Sep 29 20:32:04 dist-test-slave-p29l krb5kdc[9060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.244.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
I20260929 20:32:04.128568 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:33307
--webserver_interface=127.28.244.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:45107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:33307
--ranger_config_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.244.148:34755/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:32:04.473524 9504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:32:04.473860 9504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:32:04.474017 9504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:32:04.481784 9504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:32:04.481899 9504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:32:04.481978 9504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:32:04.482082 9504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:32:04.482164 9504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:32:04.493502 9504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.244.148:34755/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:33307
--ranger_config_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:33307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:32:04.494951 9504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:32:04.496510 9504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:32:04.510159 9509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:04.510131 9512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:04.510128 9510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:32:04.510856 9504 server_base.cc:1034] running on GCE node
I20260929 20:32:04.511760 9504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:32:04.513494 9504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:32:04.514760 9504 hybrid_clock.cc:648] HybridClock initialized: now 1790713924514717 us; error 66 us; skew 500 ppm
I20260929 20:32:04.515187 9504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:32:04 dist-test-slave-p29l krb5kdc[9060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713924, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:32:04.518751 9504 init.cc:377] Logged in from keytab as kudu/127.28.244.190@KRBTEST.COM (short username kudu)
I20260929 20:32:04.520532 9504 webserver.cc:460] Webserver started at http://127.28.244.190:36141/ using document root <none> and password file <none>
I20260929 20:32:04.521272 9504 fs_manager.cc:359] Metadata directory not provided
I20260929 20:32:04.521423 9504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:32:04.521759 9504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 20:32:04 dist-test-slave-p29l krb5kdc[9060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713924, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for HTTP/127.28.244.148@KRBTEST.COM
I20260929 20:32:04.575093 9504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "afea710921db4f3d8497301702112223"
format_stamp: "Formatted at 2026-09-29 20:32:04 on dist-test-slave-p29l"
server_key: "fd127e939c5b10b8358bb8c8f9666b12"
server_key_iv: "23af6eeaab32050f23ad6bc098bdfa93"
server_key_version: "kuduclusterkey@0"
I20260929 20:32:04.576109 9504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "afea710921db4f3d8497301702112223"
format_stamp: "Formatted at 2026-09-29 20:32:04 on dist-test-slave-p29l"
server_key: "fd127e939c5b10b8358bb8c8f9666b12"
server_key_iv: "23af6eeaab32050f23ad6bc098bdfa93"
server_key_version: "kuduclusterkey@0"
I20260929 20:32:04.581979 9504 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:32:04.599464 9519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:04.601221 9504 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 20:32:04.601456 9504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "afea710921db4f3d8497301702112223"
format_stamp: "Formatted at 2026-09-29 20:32:04 on dist-test-slave-p29l"
server_key: "fd127e939c5b10b8358bb8c8f9666b12"
server_key_iv: "23af6eeaab32050f23ad6bc098bdfa93"
server_key_version: "kuduclusterkey@0"
I20260929 20:32:04.601712 9504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:32:04.624187 9504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:32:04.628908 9504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:32:04.629253 9504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:32:04.649389 9504 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:33307
I20260929 20:32:04.649411 9570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:33307 every 8 connection(s)
I20260929 20:32:04.650950 9504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:32:04.655045 9571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:32:04.655416 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 9504
I20260929 20:32:04.655629 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:32:04.656381 29650 external_mini_cluster.cc:1428] Setting key d73854b9b6713a921fa192e2d34c4138
I20260929 20:32:04.669546 9571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223: Bootstrap starting.
Sep 29 20:32:04 dist-test-slave-p29l krb5kdc[9060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:32:04.673712 9571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223: Neither blocks nor log segments found. Creating new log.
I20260929 20:32:04.674853 9571 log.cc:824] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223: Log is configured to *not* fsync() on all Append() calls
I20260929 20:32:04.678617 9571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223: No bootstrap required, opened a new log
I20260929 20:32:04.682065 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:04.658574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55090 (local address 127.28.244.190:33307)
0929 20:32:04.659434 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:04.659449 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:04.659496 (+ 47us) server_negotiation.cc:407] Connection header received
0929 20:32:04.661217 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:04.661242 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:04.661725 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:04.662088 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:04.663871 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:04.665203 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:04.667215 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:04.668107 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:04.668262 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:04.673164 (+ 4902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:04.673174 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:04.673191 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:04.673238 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:04.677299 (+ 4061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:04.678319 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:04.678328 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:04.678335 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:04.678519 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:04.679026 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:04.679050 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:04.679052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:04.679637 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:04.679962 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:04.680428 (+ 466us) server_negotiation.cc:299] Negotiation successful
0929 20:32:04.680845 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":401,"threads_started":1}
I20260929 20:32:04.687018 9571 raft_consensus.cc:348] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afea710921db4f3d8497301702112223" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 33307 } }
I20260929 20:32:04.687413 9571 raft_consensus.cc:374] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:32:04.687516 9571 raft_consensus.cc:729] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afea710921db4f3d8497301702112223, State: Initialized, Role: FOLLOWER
I20260929 20:32:04.688220 9571 consensus_queue.cc:260] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afea710921db4f3d8497301702112223" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 33307 } }
I20260929 20:32:04.688467 9571 raft_consensus.cc:388] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:32:04.688599 9571 raft_consensus.cc:482] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:32:04.688784 9571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:32:04.691037 9571 raft_consensus.cc:504] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afea710921db4f3d8497301702112223" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 33307 } }
I20260929 20:32:04.691601 9571 leader_election.cc:302] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afea710921db4f3d8497301702112223; no voters:
I20260929 20:32:04.692224 9571 leader_election.cc:288] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:32:04.692381 9576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:32:04.693082 9576 raft_consensus.cc:686] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223 [term 1 LEADER]: Becoming Leader. State: Replica: afea710921db4f3d8497301702112223, State: Running, Role: LEADER
I20260929 20:32:04.693642 9576 consensus_queue.cc:237] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afea710921db4f3d8497301702112223" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 33307 } }
I20260929 20:32:04.694443 9571 sys_catalog.cc:564] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:32:04.696727 9577 sys_catalog.cc:455] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "afea710921db4f3d8497301702112223" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afea710921db4f3d8497301702112223" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 33307 } } }
I20260929 20:32:04.697125 9577 sys_catalog.cc:458] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223 [sys.catalog]: This master's current role is: LEADER
I20260929 20:32:04.697595 9578 sys_catalog.cc:455] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afea710921db4f3d8497301702112223. Latest consensus state: current_term: 1 leader_uuid: "afea710921db4f3d8497301702112223" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afea710921db4f3d8497301702112223" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 33307 } } }
I20260929 20:32:04.697904 9578 sys_catalog.cc:458] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223 [sys.catalog]: This master's current role is: LEADER
I20260929 20:32:04.700450 9581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:32:04.707566 9581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:32:04.713847 9571 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 20:32:04.728912 9581 catalog_manager.cc:1347] Generated new cluster ID: 37a48ad6da9645f7b3b4183588893156
I20260929 20:32:04.729075 9581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:32:04.747833 9581 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:32:04.750155 9581 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:32:04.759075 9581 catalog_manager.cc:5868] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223: Generated new TSK 0
I20260929 20:32:04.760460 9581 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.244.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab.
Entry for principal kudu/127.28.244.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab.
I20260929 20:32:05.584872 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.244.148:34755/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.129:0
--local_ip_for_outbound_sockets=127.28.244.129
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:33307
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:45107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:32:05 dist-test-slave-p29l krb5kdc[9060](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790713925, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 20:32:06.012996 9616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:32:06.013432 9616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:32:06.013650 9616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:32:06.021639 9616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:32:06.021819 9616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:32:06.022008 9616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.129
I20260929 20:32:06.033005 9616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.244.148:34755/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:33307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.129
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:32:06.034667 9616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:32:06.036533 9616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:32:06.054199 9627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:06.055011 9626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:06.055492 9629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:32:06.057320 9616 server_base.cc:1034] running on GCE node
I20260929 20:32:06.058264 9616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:32:06.059689 9616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:32:06.061079 9616 hybrid_clock.cc:648] HybridClock initialized: now 1790713926060962 us; error 106 us; skew 500 ppm
I20260929 20:32:06.061474 9616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:32:06 dist-test-slave-p29l krb5kdc[9060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713926, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:32:06.066265 9616 init.cc:377] Logged in from keytab as kudu/127.28.244.129@KRBTEST.COM (short username kudu)
I20260929 20:32:06.073019 9616 webserver.cc:460] Webserver started at http://127.28.244.129:42211/ using document root <none> and password file <none>
I20260929 20:32:06.074210 9616 fs_manager.cc:359] Metadata directory not provided
I20260929 20:32:06.074450 9616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:32:06.074905 9616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 20:32:06 dist-test-slave-p29l krb5kdc[9060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713926, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for HTTP/127.28.244.148@KRBTEST.COM
I20260929 20:32:06.097631 9616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-0/data/instance:
uuid: "9610f3ba6f514c359a49c3f5f477be22"
format_stamp: "Formatted at 2026-09-29 20:32:06 on dist-test-slave-p29l"
server_key: "0c2966956646ff42a477cdcd589d4092"
server_key_iv: "112127d1b312250bee1d97bb303ac2f7"
server_key_version: "kuduclusterkey@0"
I20260929 20:32:06.102187 9616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance:
uuid: "9610f3ba6f514c359a49c3f5f477be22"
format_stamp: "Formatted at 2026-09-29 20:32:06 on dist-test-slave-p29l"
server_key: "0c2966956646ff42a477cdcd589d4092"
server_key_iv: "112127d1b312250bee1d97bb303ac2f7"
server_key_version: "kuduclusterkey@0"
I20260929 20:32:06.111702 9616 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.007s sys 0.000s
I20260929 20:32:06.143858 9636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:06.152753 9616 fs_manager.cc:725] Time spent opening block manager: real 0.023s user 0.003s sys 0.013s
I20260929 20:32:06.153103 9616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-0/wal
uuid: "9610f3ba6f514c359a49c3f5f477be22"
format_stamp: "Formatted at 2026-09-29 20:32:06 on dist-test-slave-p29l"
server_key: "0c2966956646ff42a477cdcd589d4092"
server_key_iv: "112127d1b312250bee1d97bb303ac2f7"
server_key_version: "kuduclusterkey@0"
I20260929 20:32:06.153486 9616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:32:06.196569 9616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:32:06.201125 9616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:32:06.201474 9616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:32:06.202555 9616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:32:06.204188 9616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:32:06.204270 9616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:06.204448 9616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:32:06.204502 9616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:06.282480 9616 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.129:41821
I20260929 20:32:06.282518 9748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.129:41821 every 8 connection(s)
I20260929 20:32:06.284452 9616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
I20260929 20:32:06.291667 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 9616
I20260929 20:32:06.291968 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance
I20260929 20:32:06.292918 29650 external_mini_cluster.cc:1428] Setting key 26034cbf4c6cd5688e5de7e772b76ab8
Sep 29 20:32:06 dist-test-slave-p29l krb5kdc[9060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713926, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:32:06.319037 9751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:06.287531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:44965 (local address 127.28.244.190:33307)
0929 20:32:06.288096 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:06.288102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:06.298666 (+ 10564us) server_negotiation.cc:407] Connection header received
0929 20:32:06.298747 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:06.298752 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:06.298862 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:06.298975 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:06.301508 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:06.302682 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:06.305424 (+ 2742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:06.306197 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:06.306386 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:06.311339 (+ 4953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:06.311347 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:06.311353 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:06.311398 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:06.315101 (+ 3703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:06.316187 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:06.316193 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:06.316197 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:06.316363 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:06.316893 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:06.316899 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:06.316902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:06.317271 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:06.317401 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:06.318397 (+ 996us) server_negotiation.cc:299] Negotiation successful
0929 20:32:06.318630 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":296,"threads_started":1}
I20260929 20:32:06.321736 9752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:06.288262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:33307 (local address 127.28.244.129:44965)
0929 20:32:06.295692 (+ 7430us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:06.295746 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:06.298218 (+ 2472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:06.299498 (+ 1280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:06.299512 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:06.300308 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:32:06.301250 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:06.301283 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:06.302919 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:06.302925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:06.305232 (+ 2307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:06.305252 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:06.306744 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:06.306753 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:06.307069 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:06.307983 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:32:06.308022 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:32:06.311002 (+ 2980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:32:06.315314 (+ 4312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:06.315326 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:06.315348 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:06.316011 (+ 663us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:06.316516 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:06.316521 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:06.316525 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:06.316765 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:06.317494 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:32:06.317503 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:32:06.317959 (+ 456us) client_negotiation.cc:769] Sending connection context
0929 20:32:06.319695 (+ 1736us) client_negotiation.cc:240] Negotiation successful
0929 20:32:06.320305 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7084,"thread_start_us":6805,"threads_started":1}
I20260929 20:32:06.323967 9749 heartbeater.cc:347] Connected to a master server at 127.28.244.190:33307
I20260929 20:32:06.324307 9749 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.28.244.130@KRBTEST.COM; defaulting to no policy
I20260929 20:32:06.325181 9749 heartbeater.cc:511] Master 127.28.244.190:33307 requested a full tablet report, sending...
I20260929 20:32:06.327778 9536 ts_manager.cc:194] Registered new tserver with Master: 9610f3ba6f514c359a49c3f5f477be22 (127.28.244.129:41821)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.130@KRBTEST.COM" created.
I20260929 20:32:06.329895 9536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.129@KRBTEST.COM'} at 127.28.244.129:44965
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab.
Entry for principal kudu/127.28.244.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab.
I20260929 20:32:06.368288 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.244.148:34755/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.130:0
--local_ip_for_outbound_sockets=127.28.244.130
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:33307
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:45107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:32:06 dist-test-slave-p29l krb5kdc[9060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713925, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for HTTP/127.28.244.148@KRBTEST.COM
Sep 29 20:32:06 dist-test-slave-p29l krb5kdc[9060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713925, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for HTTP/127.28.244.148@KRBTEST.COM
W20260929 20:32:06.876865 9755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:32:06.877414 9755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:32:06.877627 9755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:32:06.889650 9755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:32:06.889915 9755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:32:06.890177 9755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.130
I20260929 20:32:06.906240 9755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.244.148:34755/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:33307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.130
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:32:06.908277 9755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:32:06.910189 9755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:32:06.942363 9762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:06.944991 9763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:06.963050 9765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:32:06.965728 9755 server_base.cc:1034] running on GCE node
I20260929 20:32:06.966691 9755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:32:06.972067 9755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:32:06.975642 9755 hybrid_clock.cc:648] HybridClock initialized: now 1790713926975507 us; error 125 us; skew 500 ppm
I20260929 20:32:06.976248 9755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:32:06 dist-test-slave-p29l krb5kdc[9060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713926, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:32:06.980928 9755 init.cc:377] Logged in from keytab as kudu/127.28.244.130@KRBTEST.COM (short username kudu)
I20260929 20:32:06.995251 9755 webserver.cc:460] Webserver started at http://127.28.244.130:36835/ using document root <none> and password file <none>
I20260929 20:32:06.996389 9755 fs_manager.cc:359] Metadata directory not provided
I20260929 20:32:06.996632 9755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:32:06.997073 9755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 20:32:06 dist-test-slave-p29l krb5kdc[9060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713926, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for HTTP/127.28.244.148@KRBTEST.COM
I20260929 20:32:07.029371 9755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-1/data/instance:
uuid: "9d4ce3c6363b495d8e7f981a2cfb1a12"
format_stamp: "Formatted at 2026-09-29 20:32:07 on dist-test-slave-p29l"
server_key: "401cf6a9616ca8ffc7e1c5b7b978db97"
server_key_iv: "f06845eaf93bacaff786045859a7b262"
server_key_version: "kuduclusterkey@0"
I20260929 20:32:07.030575 9755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance:
uuid: "9d4ce3c6363b495d8e7f981a2cfb1a12"
format_stamp: "Formatted at 2026-09-29 20:32:07 on dist-test-slave-p29l"
server_key: "401cf6a9616ca8ffc7e1c5b7b978db97"
server_key_iv: "f06845eaf93bacaff786045859a7b262"
server_key_version: "kuduclusterkey@0"
I20260929 20:32:07.040015 9755 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.010s sys 0.000s
I20260929 20:32:07.067837 9774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:07.070704 9755 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260929 20:32:07.070967 9755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-1/wal
uuid: "9d4ce3c6363b495d8e7f981a2cfb1a12"
format_stamp: "Formatted at 2026-09-29 20:32:07 on dist-test-slave-p29l"
server_key: "401cf6a9616ca8ffc7e1c5b7b978db97"
server_key_iv: "f06845eaf93bacaff786045859a7b262"
server_key_version: "kuduclusterkey@0"
I20260929 20:32:07.071271 9755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:32:07.101023 9755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:32:07.106067 9755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:32:07.106392 9755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:32:07.107596 9755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:32:07.112773 9755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:32:07.112866 9755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:07.113018 9755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:32:07.113096 9755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:07.166872 9755 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.130:33059
I20260929 20:32:07.166893 9891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.130:33059 every 8 connection(s)
I20260929 20:32:07.168406 9755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
I20260929 20:32:07.174201 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 9755
I20260929 20:32:07.174436 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance
I20260929 20:32:07.175194 29650 external_mini_cluster.cc:1428] Setting key 6a36dc834b4682d5edcbef9d9352f1bd
Sep 29 20:32:07 dist-test-slave-p29l krb5kdc[9060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713926, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:32:07.211241 9895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:07.172877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:33307 (local address 127.28.244.130:54437)
0929 20:32:07.185397 (+ 12520us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:07.185470 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:07.188420 (+ 2950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:07.189969 (+ 1549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:07.189983 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:07.190662 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:32:07.191606 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:07.191641 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:07.193369 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:07.193375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:07.195580 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:07.195609 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:07.197009 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:07.197017 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:07.197403 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:07.198270 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:32:07.198303 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:32:07.201269 (+ 2966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:32:07.205661 (+ 4392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:07.205672 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:07.205695 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:07.206486 (+ 791us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:07.207017 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:07.207023 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:07.207027 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:07.207325 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:07.208173 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:32:07.208182 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:32:07.208733 (+ 551us) client_negotiation.cc:769] Sending connection context
0929 20:32:07.209138 (+ 405us) client_negotiation.cc:240] Negotiation successful
0929 20:32:07.209754 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12049,"thread_start_us":11760,"threads_started":1}
I20260929 20:32:07.212913 9894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:07.171868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:54437 (local address 127.28.244.190:33307)
0929 20:32:07.189297 (+ 17429us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:07.189304 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:07.189323 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:32:07.189401 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:07.189406 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:07.189493 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:07.191857 (+ 2364us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:07.192029 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:07.193145 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:07.195868 (+ 2723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:07.196537 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:07.196672 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:07.201604 (+ 4932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:07.201612 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:07.201617 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:07.201672 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:07.205398 (+ 3726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:07.206686 (+ 1288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:07.206692 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:07.206696 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:07.206868 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:07.207473 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:07.207478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:07.207482 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:07.207931 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:07.208075 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:07.212166 (+ 4091us) server_negotiation.cc:299] Negotiation successful
0929 20:32:07.212510 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17279,"thread_start_us":17161,"threads_started":1}
I20260929 20:32:07.214363 9892 heartbeater.cc:347] Connected to a master server at 127.28.244.190:33307
I20260929 20:32:07.214735 9892 heartbeater.cc:464] Registering TS with master...
I20260929 20:32:07.215726 9892 heartbeater.cc:511] Master 127.28.244.190:33307 requested a full tablet report, sending...
I20260929 20:32:07.217643 9536 ts_manager.cc:194] Registered new tserver with Master: 9d4ce3c6363b495d8e7f981a2cfb1a12 (127.28.244.130:33059)
WARNING: no policy specified for kudu/127.28.244.131@KRBTEST.COM; defaulting to no policy
I20260929 20:32:07.218789 9536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.130@KRBTEST.COM'} at 127.28.244.130:54437
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab.
Entry for principal kudu/127.28.244.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab.
I20260929 20:32:07.260591 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.244.148:34755/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.131:0
--local_ip_for_outbound_sockets=127.28.244.131
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:33307
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:45107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:32:07.334018 9749 heartbeater.cc:503] Master 127.28.244.190:33307 was elected leader, sending a full tablet report...
W20260929 20:32:07.699472 9898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:32:07.699865 9898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:32:07.700048 9898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:32:07.707795 9898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:32:07.707999 9898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:32:07.708194 9898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.131
I20260929 20:32:07.718827 9898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.244.148:34755/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:33307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.131
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:32:07.720563 9898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:32:07.721967 9898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:32:07.737494 9904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:07.737591 9903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:07.737536 9906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:32:07.741683 9898 server_base.cc:1034] running on GCE node
I20260929 20:32:07.742424 9898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:32:07.743492 9898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:32:07.744793 9898 hybrid_clock.cc:648] HybridClock initialized: now 1790713927744701 us; error 79 us; skew 500 ppm
I20260929 20:32:07.745126 9898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:32:07 dist-test-slave-p29l krb5kdc[9060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713927, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:32:07.748662 9898 init.cc:377] Logged in from keytab as kudu/127.28.244.131@KRBTEST.COM (short username kudu)
I20260929 20:32:07.750363 9898 webserver.cc:460] Webserver started at http://127.28.244.131:34371/ using document root <none> and password file <none>
I20260929 20:32:07.751184 9898 fs_manager.cc:359] Metadata directory not provided
I20260929 20:32:07.751341 9898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:32:07.751693 9898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 20:32:07 dist-test-slave-p29l krb5kdc[9060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713927, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for HTTP/127.28.244.148@KRBTEST.COM
I20260929 20:32:07.769374 9898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-2/data/instance:
uuid: "85f75ac368e2472fb436ceefe945e7d8"
format_stamp: "Formatted at 2026-09-29 20:32:07 on dist-test-slave-p29l"
server_key: "7b8893d35cd72a03f15d8a9ce95f8095"
server_key_iv: "e0a6e756438367f51ec48a1eb2fee718"
server_key_version: "kuduclusterkey@0"
I20260929 20:32:07.770820 9898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance:
uuid: "85f75ac368e2472fb436ceefe945e7d8"
format_stamp: "Formatted at 2026-09-29 20:32:07 on dist-test-slave-p29l"
server_key: "7b8893d35cd72a03f15d8a9ce95f8095"
server_key_iv: "e0a6e756438367f51ec48a1eb2fee718"
server_key_version: "kuduclusterkey@0"
I20260929 20:32:07.777475 9898 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.007s
I20260929 20:32:07.795094 9913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:07.796941 9898 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 20:32:07.797189 9898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-2/wal
uuid: "85f75ac368e2472fb436ceefe945e7d8"
format_stamp: "Formatted at 2026-09-29 20:32:07 on dist-test-slave-p29l"
server_key: "7b8893d35cd72a03f15d8a9ce95f8095"
server_key_iv: "e0a6e756438367f51ec48a1eb2fee718"
server_key_version: "kuduclusterkey@0"
I20260929 20:32:07.797456 9898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:32:07.831239 9898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:32:07.836136 9898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:32:07.836527 9898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:32:07.837679 9898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:32:07.839301 9898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:32:07.839425 9898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:07.839593 9898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:32:07.839679 9898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:07.876956 9898 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.131:42687
I20260929 20:32:07.877005 10025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.131:42687 every 8 connection(s)
I20260929 20:32:07.878502 9898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
I20260929 20:32:07.884130 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 9898
I20260929 20:32:07.884322 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance
I20260929 20:32:07.884836 29650 external_mini_cluster.cc:1428] Setting key 51a2b9f976fd0029db77a0b6c375aabf
Sep 29 20:32:07 dist-test-slave-p29l krb5kdc[9060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713927, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:32:07.901641 10028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:07.881469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:56617 (local address 127.28.244.190:33307)
0929 20:32:07.881945 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:07.881949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:07.883073 (+ 1124us) server_negotiation.cc:407] Connection header received
0929 20:32:07.885155 (+ 2082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:07.885163 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:07.885295 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:07.885434 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:07.887457 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:07.888480 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:07.890251 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:07.890693 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:07.890797 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:07.894498 (+ 3701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:07.894504 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:07.894508 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:07.894556 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:07.897535 (+ 2979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:07.898342 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:07.898346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:07.898349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:07.898482 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:07.899071 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:07.899077 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:07.899080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:07.899662 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:07.899894 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:07.900854 (+ 960us) server_negotiation.cc:299] Negotiation successful
0929 20:32:07.901174 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":254,"threads_started":1}
I20260929 20:32:07.902707 10029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:07.882162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:33307 (local address 127.28.244.131:56617)
0929 20:32:07.882891 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:07.882933 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:07.884795 (+ 1862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:07.885749 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:07.885758 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:07.886446 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:32:07.887269 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:07.887303 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:07.888715 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:07.888720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:07.890090 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:07.890105 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:07.890925 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:07.890931 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:07.891200 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:07.891992 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:32:07.892018 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:32:07.894220 (+ 2202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:32:07.897708 (+ 3488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:07.897716 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:07.897733 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:07.898226 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:07.898629 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:07.898634 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:07.898637 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:07.898890 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:07.899850 (+ 960us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:32:07.899860 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:32:07.900386 (+ 526us) client_negotiation.cc:769] Sending connection context
0929 20:32:07.900697 (+ 311us) client_negotiation.cc:240] Negotiation successful
0929 20:32:07.901334 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":283,"threads_started":1}
I20260929 20:32:07.904239 10026 heartbeater.cc:347] Connected to a master server at 127.28.244.190:33307
I20260929 20:32:07.904613 10026 heartbeater.cc:464] Registering TS with master...
I20260929 20:32:07.905459 10026 heartbeater.cc:511] Master 127.28.244.190:33307 requested a full tablet report, sending...
I20260929 20:32:07.907207 9536 ts_manager.cc:194] Registered new tserver with Master: 85f75ac368e2472fb436ceefe945e7d8 (127.28.244.131:42687)
I20260929 20:32:07.908291 9536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.131@KRBTEST.COM'} at 127.28.244.131:56617
I20260929 20:32:07.914191 29650 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:32:07.915963 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 9616
Sep 29 20:32:07 dist-test-slave-p29l krb5kdc[9060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.129@KRBTEST.COM
I20260929 20:32:07.935086 10031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:07.917087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57564 (local address 127.28.244.129:41821)
0929 20:32:07.917753 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:07.917760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:07.917787 (+ 27us) server_negotiation.cc:407] Connection header received
0929 20:32:07.917849 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:07.917856 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:07.918076 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:07.918241 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:07.919790 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:07.921018 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:07.922717 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:07.923326 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:07.923527 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:07.928209 (+ 4682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:07.928227 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:07.928244 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:07.928296 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:07.931493 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:07.932542 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:07.932550 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:07.932552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:07.932754 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:07.933267 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:07.933271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:07.933273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:07.933702 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:07.933951 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:07.934439 (+ 488us) server_negotiation.cc:299] Negotiation successful
0929 20:32:07.934682 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":329,"threads_started":1}
I20260929 20:32:07.936480 9744 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:32:08.022801 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 9616
I20260929 20:32:08.039448 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 9755
Sep 29 20:32:08 dist-test-slave-p29l krb5kdc[9060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.130@KRBTEST.COM
I20260929 20:32:08.062685 10033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:08.040469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56908 (local address 127.28.244.130:33059)
0929 20:32:08.041133 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:08.041140 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:08.041170 (+ 30us) server_negotiation.cc:407] Connection header received
0929 20:32:08.041236 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:08.041243 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:08.041472 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:08.041607 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:08.042884 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:08.044213 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:08.045795 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:08.046413 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:08.046531 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:08.056001 (+ 9470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:08.056026 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:08.056049 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:08.056106 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:08.059362 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:08.060364 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:08.060372 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:08.060375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:08.060537 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:08.061030 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:08.061034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:08.061036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:08.061432 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:08.061711 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:08.061999 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 20:32:08.062221 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":366,"threads_started":1}
I20260929 20:32:08.064096 9887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:32:08.154984 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 9755
I20260929 20:32:08.172351 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 9898
Sep 29 20:32:08 dist-test-slave-p29l krb5kdc[9060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.131@KRBTEST.COM
I20260929 20:32:08.191279 10035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:08.173554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37776 (local address 127.28.244.131:42687)
0929 20:32:08.174282 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:08.174289 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:08.174320 (+ 31us) server_negotiation.cc:407] Connection header received
0929 20:32:08.174384 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:08.174391 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:08.174609 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:08.174743 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:08.176236 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:08.177307 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:08.178814 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:08.179438 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:08.179557 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:08.184771 (+ 5214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:08.184789 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:08.184807 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:08.184851 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:08.188044 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:08.188992 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:08.189002 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:08.189004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:08.189163 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:08.189578 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:08.189581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:08.189583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:08.190011 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:08.190276 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:08.190578 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 20:32:08.190849 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":378,"threads_started":1}
I20260929 20:32:08.192685 10021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:32:08.277773 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 9898
I20260929 20:32:08.294457 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 9504
I20260929 20:32:08.296063 9566 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:32:08.405614 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 9504
I20260929 20:32:08.429047 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:33307
--webserver_interface=127.28.244.190
--webserver_port=36141
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:45107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:33307
--ranger_config_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.244.148:34755/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.244.148:34755/kms with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:32:08.783694 10038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:32:08.784063 10038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:32:08.784248 10038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:32:08.791778 10038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:32:08.791996 10038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:32:08.792117 10038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:32:08.792215 10038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:32:08.792305 10038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:32:08.802928 10038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.244.148:34755/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:33307
--ranger_config_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:33307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=36141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:32:08.804471 10038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:32:08.805907 10038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:32:08.818737 10043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:08.818692 10046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:08.818650 10044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:32:08.819769 10038 server_base.cc:1034] running on GCE node
I20260929 20:32:08.820698 10038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:32:08.822440 10038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:32:08.823671 10038 hybrid_clock.cc:648] HybridClock initialized: now 1790713928823610 us; error 65 us; skew 500 ppm
I20260929 20:32:08.824095 10038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:32:08 dist-test-slave-p29l krb5kdc[9060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713928, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:32:08.827479 10038 init.cc:377] Logged in from keytab as kudu/127.28.244.190@KRBTEST.COM (short username kudu)
I20260929 20:32:08.829205 10038 webserver.cc:460] Webserver started at http://127.28.244.190:36141/ using document root <none> and password file <none>
I20260929 20:32:08.829911 10038 fs_manager.cc:359] Metadata directory not provided
I20260929 20:32:08.830029 10038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 20:32:08 dist-test-slave-p29l krb5kdc[9060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713928, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for HTTP/127.28.244.148@KRBTEST.COM
I20260929 20:32:08.887876 10038 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260929 20:32:08.891228 10054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:08.892853 10038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:32:08.893105 10038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "afea710921db4f3d8497301702112223"
format_stamp: "Formatted at 2026-09-29 20:32:04 on dist-test-slave-p29l"
server_key: "fd127e939c5b10b8358bb8c8f9666b12"
server_key_iv: "23af6eeaab32050f23ad6bc098bdfa93"
server_key_version: "kuduclusterkey@0"
I20260929 20:32:08.894151 10038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:32:08.909162 10038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:32:08.913753 10038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:32:08.914129 10038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:32:08.935317 10038 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:33307
I20260929 20:32:08.935348 10105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:33307 every 8 connection(s)
I20260929 20:32:08.937131 10038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:32:08.944551 10106 sys_catalog.cc:263] Verifying existing consensus state
I20260929 20:32:08.945359 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 10038
I20260929 20:32:08.946512 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.244.148:34755/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.129:41821
--local_ip_for_outbound_sockets=127.28.244.129
--tserver_master_addrs=127.28.244.190:33307
--webserver_port=42211
--webserver_interface=127.28.244.129
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:45107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.244.148:34755/kms with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:32:08.950106 10106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223: Bootstrap starting.
I20260929 20:32:08.976670 10106 log.cc:824] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223: Log is configured to *not* fsync() on all Append() calls
I20260929 20:32:08.988736 10106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:32:08.989430 10106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223: Bootstrap complete.
I20260929 20:32:09.002699 10106 raft_consensus.cc:348] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afea710921db4f3d8497301702112223" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 33307 } }
I20260929 20:32:09.003999 10106 raft_consensus.cc:729] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: afea710921db4f3d8497301702112223, State: Initialized, Role: FOLLOWER
I20260929 20:32:09.004865 10106 consensus_queue.cc:260] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afea710921db4f3d8497301702112223" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 33307 } }
I20260929 20:32:09.005208 10106 raft_consensus.cc:388] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:32:09.005354 10106 raft_consensus.cc:482] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:32:09.005545 10106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223 [term 1 FOLLOWER]: Advancing to term 2
I20260929 20:32:09.011781 10106 raft_consensus.cc:504] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afea710921db4f3d8497301702112223" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 33307 } }
I20260929 20:32:09.012598 10106 leader_election.cc:302] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afea710921db4f3d8497301702112223; no voters:
I20260929 20:32:09.013271 10106 leader_election.cc:288] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 20:32:09.013447 10110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 20:32:09.014854 10110 raft_consensus.cc:686] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223 [term 2 LEADER]: Becoming Leader. State: Replica: afea710921db4f3d8497301702112223, State: Running, Role: LEADER
I20260929 20:32:09.015617 10106 sys_catalog.cc:564] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:32:09.015501 10110 consensus_queue.cc:237] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afea710921db4f3d8497301702112223" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 33307 } }
I20260929 20:32:09.018314 10111 sys_catalog.cc:455] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "afea710921db4f3d8497301702112223" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afea710921db4f3d8497301702112223" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 33307 } } }
I20260929 20:32:09.018311 10112 sys_catalog.cc:455] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afea710921db4f3d8497301702112223. Latest consensus state: current_term: 2 leader_uuid: "afea710921db4f3d8497301702112223" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afea710921db4f3d8497301702112223" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 33307 } } }
I20260929 20:32:09.018797 10112 sys_catalog.cc:458] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223 [sys.catalog]: This master's current role is: LEADER
I20260929 20:32:09.018795 10111 sys_catalog.cc:458] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223 [sys.catalog]: This master's current role is: LEADER
I20260929 20:32:09.030905 10115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:32:09.037686 10115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:32:09.039769 10115 catalog_manager.cc:1259] Loaded cluster ID: 37a48ad6da9645f7b3b4183588893156
I20260929 20:32:09.040021 10115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:32:09.044495 10115 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:32:09.046331 10115 catalog_manager.cc:5879] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223: Loaded TSK: 0
I20260929 20:32:09.048278 10115 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:32:09.053887 10106 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260929 20:32:09.427379 10108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:32:09.427762 10108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:32:09.427953 10108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:32:09.436818 10108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:32:09.436986 10108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:32:09.437171 10108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.129
I20260929 20:32:09.449630 10108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.244.148:34755/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.129:41821
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.244.129
--webserver_port=42211
--tserver_master_addrs=127.28.244.190:33307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.129
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:32:09.451678 10108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:32:09.453634 10108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:32:09.505165 10143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:09.505920 10144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:32:09.514555 10108 server_base.cc:1034] running on GCE node
W20260929 20:32:09.514873 10146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:32:09.515645 10108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:32:09.517086 10108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:32:09.518404 10108 hybrid_clock.cc:648] HybridClock initialized: now 1790713929518347 us; error 74 us; skew 500 ppm
I20260929 20:32:09.518858 10108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:32:09 dist-test-slave-p29l krb5kdc[9060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713929, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:32:09.527458 10108 init.cc:377] Logged in from keytab as kudu/127.28.244.129@KRBTEST.COM (short username kudu)
I20260929 20:32:09.532303 10108 webserver.cc:460] Webserver started at http://127.28.244.129:42211/ using document root <none> and password file <none>
I20260929 20:32:09.533396 10108 fs_manager.cc:359] Metadata directory not provided
I20260929 20:32:09.533598 10108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 20:32:09 dist-test-slave-p29l krb5kdc[9060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713929, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for HTTP/127.28.244.148@KRBTEST.COM
I20260929 20:32:09.579123 10108 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.007s sys 0.004s
I20260929 20:32:09.584453 10154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:09.586161 10108 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.004s
I20260929 20:32:09.586382 10108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-0/wal
uuid: "9610f3ba6f514c359a49c3f5f477be22"
format_stamp: "Formatted at 2026-09-29 20:32:06 on dist-test-slave-p29l"
server_key: "0c2966956646ff42a477cdcd589d4092"
server_key_iv: "112127d1b312250bee1d97bb303ac2f7"
server_key_version: "kuduclusterkey@0"
I20260929 20:32:09.587641 10108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:32:09.624905 10108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:32:09.630870 10108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:32:09.631305 10108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:32:09.634776 10108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:32:09.637084 10108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:32:09.637231 10108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:09.637428 10108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:32:09.637539 10108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:09.718627 10108 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.129:41821
I20260929 20:32:09.718791 10267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.129:41821 every 8 connection(s)
I20260929 20:32:09.720710 10108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
I20260929 20:32:09.722931 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 10108
I20260929 20:32:09.723917 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.244.148:34755/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.130:33059
--local_ip_for_outbound_sockets=127.28.244.130
--tserver_master_addrs=127.28.244.190:33307
--webserver_port=36835
--webserver_interface=127.28.244.130
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:45107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.244.148:34755/kms with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:32:09 dist-test-slave-p29l krb5kdc[9060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713929, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:32:09.770097 10271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:09.726509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:41953 (local address 127.28.244.190:33307)
0929 20:32:09.735217 (+ 8708us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:09.735243 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:09.740556 (+ 5313us) server_negotiation.cc:407] Connection header received
0929 20:32:09.744218 (+ 3662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:09.744246 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:09.744886 (+ 640us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:09.745391 (+ 505us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:09.747501 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:09.749581 (+ 2080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:09.752295 (+ 2714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:09.753620 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:09.753899 (+ 279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:09.759567 (+ 5668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:09.759585 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:09.759612 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:09.759685 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:09.763995 (+ 4310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:09.765148 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:09.765159 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:09.765168 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:09.765387 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:09.765965 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:09.765970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:09.765974 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:09.766687 (+ 713us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:09.766898 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:09.768163 (+ 1265us) server_negotiation.cc:299] Negotiation successful
0929 20:32:09.768654 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8380,"thread_start_us":8145,"threads_started":1}
I20260929 20:32:09.772672 10270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:09.725266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:33307 (local address 127.28.244.129:41953)
0929 20:32:09.740301 (+ 15035us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:09.740358 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:09.743287 (+ 2929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:09.745587 (+ 2300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:09.745601 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:09.746351 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:32:09.747210 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:09.747241 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:09.749853 (+ 2612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:09.749860 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:09.752079 (+ 2219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:09.752099 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:09.754371 (+ 2272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:09.754382 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:09.754785 (+ 403us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:09.755686 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:32:09.755726 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:32:09.759110 (+ 3384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:32:09.764251 (+ 5141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:09.764263 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:09.764283 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:09.764948 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:09.765532 (+ 584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:09.765537 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:09.765541 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:09.765831 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:09.766986 (+ 1155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:32:09.766996 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:32:09.767434 (+ 438us) client_negotiation.cc:769] Sending connection context
0929 20:32:09.767752 (+ 318us) client_negotiation.cc:240] Negotiation successful
0929 20:32:09.771135 (+ 3383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14706,"thread_start_us":10664,"threads_started":1}
I20260929 20:32:09.777102 10268 heartbeater.cc:347] Connected to a master server at 127.28.244.190:33307
I20260929 20:32:09.777472 10268 heartbeater.cc:464] Registering TS with master...
I20260929 20:32:09.778363 10268 heartbeater.cc:511] Master 127.28.244.190:33307 requested a full tablet report, sending...
W20260929 20:32:09.781939 10268 heartbeater.cc:644] Failed to heartbeat to 127.28.244.190:33307 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 20:32:09.782197 10268 heartbeater.cc:464] Registering TS with master...
I20260929 20:32:09.782541 10268 heartbeater.cc:511] Master 127.28.244.190:33307 requested a full tablet report, sending...
I20260929 20:32:09.784113 10268 heartbeater.cc:464] Registering TS with master...
I20260929 20:32:09.784435 10268 heartbeater.cc:511] Master 127.28.244.190:33307 requested a full tablet report, sending...
W20260929 20:32:09.785617 10268 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 20:32:10.150693 10272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:32:10.151119 10272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:32:10.151324 10272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:32:10.161995 10272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:32:10.162169 10272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:32:10.162354 10272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.130
I20260929 20:32:10.175196 10272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.244.148:34755/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.130:33059
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.244.130
--webserver_port=36835
--tserver_master_addrs=127.28.244.190:33307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.130
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:32:10.177024 10272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:32:10.178869 10272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:32:10.197013 10287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:10.197642 10286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:10.198946 10289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:32:10.200183 10272 server_base.cc:1034] running on GCE node
I20260929 20:32:10.200887 10272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:32:10.202104 10272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:32:10.203343 10272 hybrid_clock.cc:648] HybridClock initialized: now 1790713930203270 us; error 71 us; skew 500 ppm
I20260929 20:32:10.203708 10272 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:32:10 dist-test-slave-p29l krb5kdc[9060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713930, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:32:10.207335 10272 init.cc:377] Logged in from keytab as kudu/127.28.244.130@KRBTEST.COM (short username kudu)
I20260929 20:32:10.211902 10272 webserver.cc:460] Webserver started at http://127.28.244.130:36835/ using document root <none> and password file <none>
I20260929 20:32:10.212809 10272 fs_manager.cc:359] Metadata directory not provided
I20260929 20:32:10.212978 10272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 20:32:10 dist-test-slave-p29l krb5kdc[9060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713930, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for HTTP/127.28.244.148@KRBTEST.COM
I20260929 20:32:10.238615 10272 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:32:10.243356 10297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:10.245504 10272 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260929 20:32:10.245729 10272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-1/wal
uuid: "9d4ce3c6363b495d8e7f981a2cfb1a12"
format_stamp: "Formatted at 2026-09-29 20:32:07 on dist-test-slave-p29l"
server_key: "401cf6a9616ca8ffc7e1c5b7b978db97"
server_key_iv: "f06845eaf93bacaff786045859a7b262"
server_key_version: "kuduclusterkey@0"
I20260929 20:32:10.247229 10272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:32:10.296202 10272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:32:10.302520 10272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:32:10.302922 10272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:32:10.308549 10272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:32:10.310434 10272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:32:10.310544 10272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:10.310674 10272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:32:10.310746 10272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:10.368605 10272 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.130:33059
I20260929 20:32:10.368678 10411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.130:33059 every 8 connection(s)
I20260929 20:32:10.371109 10272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
I20260929 20:32:10.380735 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 10272
I20260929 20:32:10.381632 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.244.148:34755/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.131:42687
--local_ip_for_outbound_sockets=127.28.244.131
--tserver_master_addrs=127.28.244.190:33307
--webserver_port=34371
--webserver_interface=127.28.244.131
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:45107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.244.148:34755/kms with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:32:10 dist-test-slave-p29l krb5kdc[9060](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790713930, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 20:32:10 dist-test-slave-p29l krb5kdc[9060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713930, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:32:10.439904 10414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:10.375306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:33307 (local address 127.28.244.130:34679)
0929 20:32:10.408305 (+ 32999us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:10.408470 (+ 165us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:10.411222 (+ 2752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:10.413619 (+ 2397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:10.413635 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:10.414360 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:32:10.415357 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:10.415391 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:10.417719 (+ 2328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:10.417726 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:10.420060 (+ 2334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:10.420094 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:10.421588 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:10.421598 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:10.421942 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:10.424062 (+ 2120us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:32:10.424098 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:32:10.429342 (+ 5244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:32:10.434222 (+ 4880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:10.434235 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:10.434258 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:10.435020 (+ 762us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:10.435641 (+ 621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:10.435647 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:10.435651 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:10.436004 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:10.436827 (+ 823us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:32:10.436837 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:32:10.437343 (+ 506us) client_negotiation.cc:769] Sending connection context
0929 20:32:10.437681 (+ 338us) client_negotiation.cc:240] Negotiation successful
0929 20:32:10.438373 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32578,"thread_start_us":36877,"threads_started":1}
I20260929 20:32:10.441594 10415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:10.376105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:34679 (local address 127.28.244.190:33307)
0929 20:32:10.407584 (+ 31479us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:10.407591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:10.411860 (+ 4269us) server_negotiation.cc:407] Connection header received
0929 20:32:10.411955 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:10.411961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:10.412066 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:10.412186 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:10.415718 (+ 3532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:10.417401 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:10.420317 (+ 2916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:10.421034 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:10.421171 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:10.429688 (+ 8517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:10.429696 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:10.429700 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:10.429754 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:10.433932 (+ 4178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:10.435242 (+ 1310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:10.435249 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:10.435253 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:10.435438 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:10.436175 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:10.436180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:10.436184 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:10.436578 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:10.436732 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:10.440868 (+ 4136us) server_negotiation.cc:299] Negotiation successful
0929 20:32:10.441198 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31316,"thread_start_us":31067,"threads_started":1}
I20260929 20:32:10.446249 10412 heartbeater.cc:347] Connected to a master server at 127.28.244.190:33307
I20260929 20:32:10.446640 10412 heartbeater.cc:464] Registering TS with master...
I20260929 20:32:10.447660 10412 heartbeater.cc:511] Master 127.28.244.190:33307 requested a full tablet report, sending...
I20260929 20:32:10.450234 10071 ts_manager.cc:194] Registered new tserver with Master: 9d4ce3c6363b495d8e7f981a2cfb1a12 (127.28.244.130:33059)
I20260929 20:32:10.452692 10071 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.130@KRBTEST.COM'} at 127.28.244.130:34679
I20260929 20:32:10.785815 10268 heartbeater.cc:464] Registering TS with master...
I20260929 20:32:10.786299 10268 heartbeater.cc:511] Master 127.28.244.190:33307 requested a full tablet report, sending...
I20260929 20:32:10.789386 10071 ts_manager.cc:194] Registered new tserver with Master: 9610f3ba6f514c359a49c3f5f477be22 (127.28.244.129:41821)
I20260929 20:32:10.790647 10071 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.129@KRBTEST.COM'} at 127.28.244.129:41953
Sep 29 20:32:10 dist-test-slave-p29l krb5kdc[9060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713930, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for HTTP/127.28.244.148@KRBTEST.COM
W20260929 20:32:10.893496 10416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:32:10.894045 10416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:32:10.894301 10416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:32:10.906075 10416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:32:10.906349 10416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:32:10.906610 10416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.131
I20260929 20:32:10.923101 10416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.244.148:34755/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.131:42687
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.244.131
--webserver_port=34371
--tserver_master_addrs=127.28.244.190:33307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.131
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:32:10.925266 10416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:32:10.927083 10416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:32:10.988416 10423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:10.991483 10422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:32:10.992194 10416 server_base.cc:1034] running on GCE node
W20260929 20:32:10.992589 10426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:32:10.993584 10416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:32:10.999584 10416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:32:11.001202 10416 hybrid_clock.cc:648] HybridClock initialized: now 1790713931001079 us; error 126 us; skew 500 ppm
I20260929 20:32:11.001749 10416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:32:11 dist-test-slave-p29l krb5kdc[9060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713931, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:32:11.006387 10416 init.cc:377] Logged in from keytab as kudu/127.28.244.131@KRBTEST.COM (short username kudu)
I20260929 20:32:11.014642 10416 webserver.cc:460] Webserver started at http://127.28.244.131:34371/ using document root <none> and password file <none>
I20260929 20:32:11.015487 10416 fs_manager.cc:359] Metadata directory not provided
I20260929 20:32:11.015677 10416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 20:32:11 dist-test-slave-p29l krb5kdc[9060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713931, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for HTTP/127.28.244.148@KRBTEST.COM
I20260929 20:32:11.042059 10416 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.004s sys 0.004s
I20260929 20:32:11.045946 10433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:11.047933 10416 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260929 20:32:11.048131 10416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-2/wal
uuid: "85f75ac368e2472fb436ceefe945e7d8"
format_stamp: "Formatted at 2026-09-29 20:32:07 on dist-test-slave-p29l"
server_key: "7b8893d35cd72a03f15d8a9ce95f8095"
server_key_iv: "e0a6e756438367f51ec48a1eb2fee718"
server_key_version: "kuduclusterkey@0"
I20260929 20:32:11.049628 10416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:32:11.135077 10416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:32:11.141120 10416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:32:11.141522 10416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:32:11.157644 10416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:32:11.173713 10416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:32:11.173940 10416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260929 20:32:11.174120 10416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:32:11.174232 10416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:32:11 dist-test-slave-p29l krb5kdc[9060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713930, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for HTTP/127.28.244.148@KRBTEST.COM
I20260929 20:32:11.457911 10412 heartbeater.cc:503] Master 127.28.244.190:33307 was elected leader, sending a full tablet report...
I20260929 20:32:11.505366 10416 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.131:42687
I20260929 20:32:11.505439 10547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.131:42687 every 8 connection(s)
I20260929 20:32:11.507489 10416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
I20260929 20:32:11.508015 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 10416
I20260929 20:32:11.549134 10550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:11.509114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48668 (local address 127.28.244.190:33307)
0929 20:32:11.527161 (+ 18047us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:11.527167 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:11.527198 (+ 31us) server_negotiation.cc:407] Connection header received
0929 20:32:11.527299 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:11.527308 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:11.527417 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:11.527571 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:11.529718 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:11.530867 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:11.533068 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:11.533745 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:11.533878 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:11.538290 (+ 4412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:11.538299 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:11.538304 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:11.538355 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:11.544288 (+ 5933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:11.545316 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:11.545322 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:11.545327 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:11.545467 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:11.545906 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:11.545918 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:11.545921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:11.546268 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:11.546526 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:11.548482 (+ 1956us) server_negotiation.cc:299] Negotiation successful
0929 20:32:11.548738 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17892,"thread_start_us":18582,"threads_started":1}
Sep 29 20:32:11 dist-test-slave-p29l krb5kdc[9060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713931, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:32:11.569656 10553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:11.527920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:42017 (local address 127.28.244.190:33307)
0929 20:32:11.537002 (+ 9082us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:11.537008 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:11.539131 (+ 2123us) server_negotiation.cc:407] Connection header received
0929 20:32:11.544545 (+ 5414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:11.544550 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:11.544653 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:11.544740 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:11.550638 (+ 5898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:11.553098 (+ 2460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:11.555976 (+ 2878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:11.556713 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:11.556954 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:11.561968 (+ 5014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:11.561977 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:11.561982 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:11.562029 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:11.565898 (+ 3869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:11.566916 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:11.566922 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:11.566926 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:11.567100 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:11.567628 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:11.567634 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:11.567637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:11.568073 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:11.568230 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:11.569003 (+ 773us) server_negotiation.cc:299] Negotiation successful
0929 20:32:11.569262 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8921,"thread_start_us":9100,"threads_started":1}
I20260929 20:32:11.570694 10552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:11.523938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:33307 (local address 127.28.244.131:42017)
0929 20:32:11.537788 (+ 13850us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:11.537864 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:11.542455 (+ 4591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:11.546567 (+ 4112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:11.546588 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:11.547288 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:32:11.548297 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:11.550412 (+ 2115us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:11.553281 (+ 2869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:11.553288 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:11.555687 (+ 2399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:11.555747 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:11.557378 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:11.557388 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:11.557734 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:11.558641 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:32:11.558676 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:32:11.561613 (+ 2937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:32:11.566136 (+ 4523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:11.566148 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:11.566172 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:11.566747 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:11.567243 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:11.567247 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:11.567249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:11.567477 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:11.568219 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:32:11.568226 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:32:11.568597 (+ 371us) client_negotiation.cc:769] Sending connection context
0929 20:32:11.568839 (+ 242us) client_negotiation.cc:240] Negotiation successful
0929 20:32:11.569389 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13459,"thread_start_us":14093,"threads_started":1}
I20260929 20:32:11.572345 10548 heartbeater.cc:347] Connected to a master server at 127.28.244.190:33307
I20260929 20:32:11.572706 10548 heartbeater.cc:464] Registering TS with master...
I20260929 20:32:11.573566 10548 heartbeater.cc:511] Master 127.28.244.190:33307 requested a full tablet report, sending...
I20260929 20:32:11.575488 10071 ts_manager.cc:194] Registered new tserver with Master: 85f75ac368e2472fb436ceefe945e7d8 (127.28.244.131:42687)
I20260929 20:32:11.576714 10071 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.131@KRBTEST.COM'} at 127.28.244.131:42017
I20260929 20:32:11.580334 29650 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:32:11.604276 10553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:11.584071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48682 (local address 127.28.244.190:33307)
0929 20:32:11.584327 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:11.584337 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:11.584779 (+ 442us) server_negotiation.cc:407] Connection header received
0929 20:32:11.584967 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:11.584971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:11.585050 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:11.585146 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:11.587531 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:11.588500 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:11.590609 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:11.591076 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:11.591212 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:11.595186 (+ 3974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:11.595192 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:11.595195 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:11.595237 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:11.599725 (+ 4488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:11.600841 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:11.600846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:11.600850 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:11.601039 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:11.601570 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:11.601575 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:11.601578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:11.601963 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:11.603382 (+ 1419us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:11.603549 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 20:32:11.603874 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 20:32:11.611526 10071 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48682:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:32:11.623416 10071 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:32:11.652462 10570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:11.641124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:33059 (local address 127.0.0.1:56912)
0929 20:32:11.642485 (+ 1361us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:11.642523 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:11.642736 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:11.643778 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:11.643786 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:11.643843 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:32:11.644274 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:11.644287 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:11.646265 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:11.646271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:11.648789 (+ 2518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:11.648813 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:11.651275 (+ 2462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:11.651283 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:11.651554 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:11.651618 (+ 64us) client_negotiation.cc:769] Sending connection context
0929 20:32:11.651784 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 20:32:11.652037 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1085,"thread_start_us":1117,"threads_started":1}
I20260929 20:32:11.653498 10572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:11.641368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56912 (local address 127.28.244.130:33059)
0929 20:32:11.642070 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:11.642085 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:11.642648 (+ 563us) server_negotiation.cc:407] Connection header received
0929 20:32:11.642986 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:11.642995 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:11.643278 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:11.643475 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:32:11.644505 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:11.646071 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:11.649049 (+ 2978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:11.650620 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:11.650865 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:11.651072 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:11.652919 (+ 1847us) server_negotiation.cc:299] Negotiation successful
0929 20:32:11.653169 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":312,"threads_started":1}
I20260929 20:32:11.662281 10347 tablet_service.cc:1467] Processing CreateTablet for tablet 7f8b6656824d444f8b04f6d0fed1dcf7 (DEFAULT_TABLE table=test-table [id=61f8dc4a251c4f0abde48ae68167f9a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:32:11.664888 10347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f8b6656824d444f8b04f6d0fed1dcf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:32:11.664999 10570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:11.643123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:41821 (local address 127.0.0.1:57570)
0929 20:32:11.654081 (+ 10958us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:11.654101 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:11.654308 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:11.656395 (+ 2087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:11.656402 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:11.656438 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:32:11.656634 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:11.656646 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:11.659784 (+ 3138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:11.659791 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:11.661728 (+ 1937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:11.661748 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:11.664103 (+ 2355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:11.664111 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:11.664320 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:11.664356 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 20:32:11.664464 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 20:32:11.664659 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10848,"thread_start_us":13366,"threads_started":1}
I20260929 20:32:11.665748 10574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:11.643490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57570 (local address 127.28.244.129:41821)
0929 20:32:11.655621 (+ 12131us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:11.655628 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:11.655654 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:32:11.655743 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:11.655749 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:11.656104 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:11.656297 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:32:11.656822 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:11.659525 (+ 2703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:11.661960 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:11.663432 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:11.663739 (+ 307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:11.663972 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:11.665202 (+ 1230us) server_negotiation.cc:299] Negotiation successful
0929 20:32:11.665393 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11952,"thread_start_us":11794,"threads_started":1}
I20260929 20:32:11.670795 10203 tablet_service.cc:1467] Processing CreateTablet for tablet 7f8b6656824d444f8b04f6d0fed1dcf7 (DEFAULT_TABLE table=test-table [id=61f8dc4a251c4f0abde48ae68167f9a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:32:11.672477 10203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f8b6656824d444f8b04f6d0fed1dcf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:32:11.677043 10573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:11.656955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:42687 (local address 127.0.0.1:37786)
0929 20:32:11.658201 (+ 1246us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:11.658221 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:11.658336 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:11.668645 (+ 10309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:11.668651 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:11.668682 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:32:11.669029 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:11.669041 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:11.671001 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:11.671010 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:11.673635 (+ 2625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:11.673657 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:11.676140 (+ 2483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:11.676148 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:11.676334 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:11.676373 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 20:32:11.676477 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 20:32:11.676657 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1106}
I20260929 20:32:11.677947 10575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:11.657307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37786 (local address 127.28.244.131:42687)
0929 20:32:11.667616 (+ 10309us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:11.667626 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:11.667658 (+ 32us) server_negotiation.cc:407] Connection header received
0929 20:32:11.667759 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:11.667766 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:11.668063 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:11.668271 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:32:11.669236 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:11.670804 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:11.673895 (+ 3091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:11.675495 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:11.675733 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:11.675961 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:11.677390 (+ 1429us) server_negotiation.cc:299] Negotiation successful
0929 20:32:11.677593 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10087,"thread_start_us":9135,"threads_started":1}
I20260929 20:32:11.682526 10482 tablet_service.cc:1467] Processing CreateTablet for tablet 7f8b6656824d444f8b04f6d0fed1dcf7 (DEFAULT_TABLE table=test-table [id=61f8dc4a251c4f0abde48ae68167f9a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:32:11.684464 10482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f8b6656824d444f8b04f6d0fed1dcf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:32:11.699515 10576 tablet_bootstrap.cc:492] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9d4ce3c6363b495d8e7f981a2cfb1a12: Bootstrap starting.
I20260929 20:32:11.708003 10576 tablet_bootstrap.cc:654] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9d4ce3c6363b495d8e7f981a2cfb1a12: Neither blocks nor log segments found. Creating new log.
I20260929 20:32:11.709443 10576 log.cc:824] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9d4ce3c6363b495d8e7f981a2cfb1a12: Log is configured to *not* fsync() on all Append() calls
I20260929 20:32:11.710770 10577 tablet_bootstrap.cc:492] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9610f3ba6f514c359a49c3f5f477be22: Bootstrap starting.
I20260929 20:32:11.712267 10578 tablet_bootstrap.cc:492] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 85f75ac368e2472fb436ceefe945e7d8: Bootstrap starting.
I20260929 20:32:11.714751 10577 tablet_bootstrap.cc:654] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9610f3ba6f514c359a49c3f5f477be22: Neither blocks nor log segments found. Creating new log.
I20260929 20:32:11.715073 10578 tablet_bootstrap.cc:654] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 85f75ac368e2472fb436ceefe945e7d8: Neither blocks nor log segments found. Creating new log.
I20260929 20:32:11.716421 10578 log.cc:824] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 85f75ac368e2472fb436ceefe945e7d8: Log is configured to *not* fsync() on all Append() calls
I20260929 20:32:11.716485 10577 log.cc:824] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9610f3ba6f514c359a49c3f5f477be22: Log is configured to *not* fsync() on all Append() calls
I20260929 20:32:11.719859 10576 tablet_bootstrap.cc:492] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9d4ce3c6363b495d8e7f981a2cfb1a12: No bootstrap required, opened a new log
I20260929 20:32:11.720248 10576 ts_tablet_manager.cc:1397] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9d4ce3c6363b495d8e7f981a2cfb1a12: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.000s
I20260929 20:32:11.720784 10578 tablet_bootstrap.cc:492] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 85f75ac368e2472fb436ceefe945e7d8: No bootstrap required, opened a new log
I20260929 20:32:11.721056 10578 ts_tablet_manager.cc:1397] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 85f75ac368e2472fb436ceefe945e7d8: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260929 20:32:11.731259 10576 raft_consensus.cc:348] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9d4ce3c6363b495d8e7f981a2cfb1a12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d4ce3c6363b495d8e7f981a2cfb1a12" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33059 } } peers { permanent_uuid: "9610f3ba6f514c359a49c3f5f477be22" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 41821 } } peers { permanent_uuid: "85f75ac368e2472fb436ceefe945e7d8" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 42687 } }
I20260929 20:32:11.732057 10576 raft_consensus.cc:374] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9d4ce3c6363b495d8e7f981a2cfb1a12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:32:11.732260 10576 raft_consensus.cc:729] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9d4ce3c6363b495d8e7f981a2cfb1a12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d4ce3c6363b495d8e7f981a2cfb1a12, State: Initialized, Role: FOLLOWER
I20260929 20:32:11.733258 10577 tablet_bootstrap.cc:492] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9610f3ba6f514c359a49c3f5f477be22: No bootstrap required, opened a new log
I20260929 20:32:11.733062 10578 raft_consensus.cc:348] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 85f75ac368e2472fb436ceefe945e7d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d4ce3c6363b495d8e7f981a2cfb1a12" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33059 } } peers { permanent_uuid: "9610f3ba6f514c359a49c3f5f477be22" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 41821 } } peers { permanent_uuid: "85f75ac368e2472fb436ceefe945e7d8" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 42687 } }
I20260929 20:32:11.733230 10576 consensus_queue.cc:260] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9d4ce3c6363b495d8e7f981a2cfb1a12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d4ce3c6363b495d8e7f981a2cfb1a12" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33059 } } peers { permanent_uuid: "9610f3ba6f514c359a49c3f5f477be22" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 41821 } } peers { permanent_uuid: "85f75ac368e2472fb436ceefe945e7d8" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 42687 } }
I20260929 20:32:11.733613 10578 raft_consensus.cc:374] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 85f75ac368e2472fb436ceefe945e7d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:32:11.733629 10577 ts_tablet_manager.cc:1397] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9610f3ba6f514c359a49c3f5f477be22: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.000s
I20260929 20:32:11.733776 10578 raft_consensus.cc:729] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 85f75ac368e2472fb436ceefe945e7d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f75ac368e2472fb436ceefe945e7d8, State: Initialized, Role: FOLLOWER
I20260929 20:32:11.734596 10578 consensus_queue.cc:260] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 85f75ac368e2472fb436ceefe945e7d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d4ce3c6363b495d8e7f981a2cfb1a12" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33059 } } peers { permanent_uuid: "9610f3ba6f514c359a49c3f5f477be22" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 41821 } } peers { permanent_uuid: "85f75ac368e2472fb436ceefe945e7d8" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 42687 } }
I20260929 20:32:11.742295 10576 ts_tablet_manager.cc:1428] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9d4ce3c6363b495d8e7f981a2cfb1a12: Time spent starting tablet: real 0.022s user 0.013s sys 0.006s
I20260929 20:32:11.742877 10578 ts_tablet_manager.cc:1428] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 85f75ac368e2472fb436ceefe945e7d8: Time spent starting tablet: real 0.022s user 0.011s sys 0.008s
I20260929 20:32:11.743336 10548 heartbeater.cc:503] Master 127.28.244.190:33307 was elected leader, sending a full tablet report...
I20260929 20:32:11.747741 10577 raft_consensus.cc:348] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9610f3ba6f514c359a49c3f5f477be22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d4ce3c6363b495d8e7f981a2cfb1a12" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33059 } } peers { permanent_uuid: "9610f3ba6f514c359a49c3f5f477be22" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 41821 } } peers { permanent_uuid: "85f75ac368e2472fb436ceefe945e7d8" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 42687 } }
I20260929 20:32:11.748486 10577 raft_consensus.cc:374] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9610f3ba6f514c359a49c3f5f477be22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:32:11.748664 10577 raft_consensus.cc:729] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9610f3ba6f514c359a49c3f5f477be22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9610f3ba6f514c359a49c3f5f477be22, State: Initialized, Role: FOLLOWER
I20260929 20:32:11.749589 10577 consensus_queue.cc:260] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9610f3ba6f514c359a49c3f5f477be22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d4ce3c6363b495d8e7f981a2cfb1a12" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33059 } } peers { permanent_uuid: "9610f3ba6f514c359a49c3f5f477be22" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 41821 } } peers { permanent_uuid: "85f75ac368e2472fb436ceefe945e7d8" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 42687 } }
I20260929 20:32:11.750873 10268 heartbeater.cc:503] Master 127.28.244.190:33307 was elected leader, sending a full tablet report...
I20260929 20:32:11.751561 10577 ts_tablet_manager.cc:1428] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9610f3ba6f514c359a49c3f5f477be22: Time spent starting tablet: real 0.018s user 0.016s sys 0.000s
I20260929 20:32:11.763787 10584 raft_consensus.cc:482] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9610f3ba6f514c359a49c3f5f477be22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:32:11.764214 10584 raft_consensus.cc:504] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9610f3ba6f514c359a49c3f5f477be22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d4ce3c6363b495d8e7f981a2cfb1a12" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33059 } } peers { permanent_uuid: "9610f3ba6f514c359a49c3f5f477be22" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 41821 } } peers { permanent_uuid: "85f75ac368e2472fb436ceefe945e7d8" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 42687 } }
I20260929 20:32:11.766633 10584 leader_election.cc:288] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9610f3ba6f514c359a49c3f5f477be22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d4ce3c6363b495d8e7f981a2cfb1a12 (127.28.244.130:33059), 85f75ac368e2472fb436ceefe945e7d8 (127.28.244.131:42687)
W20260929 20:32:11.775331 10551 tablet.cc:2369] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 85f75ac368e2472fb436ceefe945e7d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:32:11.775837 10575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:11.768337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:45907 (local address 127.28.244.131:42687)
0929 20:32:11.768517 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:11.768523 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:11.768852 (+ 329us) server_negotiation.cc:407] Connection header received
0929 20:32:11.769067 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:11.769072 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:11.769150 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:11.769249 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:32:11.769972 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:11.771055 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:11.773757 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:11.774635 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:11.774846 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:11.774938 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:11.775335 (+ 397us) server_negotiation.cc:299] Negotiation successful
0929 20:32:11.775509 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 20:32:11.776181 10572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:11.767322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:57353 (local address 127.28.244.130:33059)
0929 20:32:11.767523 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:11.767528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:11.767956 (+ 428us) server_negotiation.cc:407] Connection header received
0929 20:32:11.768524 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:11.768531 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:11.768619 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:11.768678 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:32:11.769510 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:11.770516 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:11.773497 (+ 2981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:11.774880 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:11.775088 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:11.775175 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:11.775741 (+ 566us) server_negotiation.cc:299] Negotiation successful
0929 20:32:11.775919 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 20:32:11.776842 10586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:11.768105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:42687 (local address 127.28.244.129:45907)
0929 20:32:11.768743 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:11.768759 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:11.768927 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:11.769287 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:11.769292 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:11.769322 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:32:11.769781 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:11.769794 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:11.771214 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:11.771221 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:11.773586 (+ 2365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:11.773598 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:11.774885 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:11.774892 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:11.775102 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:11.775138 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 20:32:11.776146 (+ 1008us) client_negotiation.cc:240] Negotiation successful
0929 20:32:11.776330 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"mutex_wait_us":21,"thread_start_us":332,"threads_started":1}
I20260929 20:32:11.777007 10585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:11.767089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:33059 (local address 127.28.244.129:57353)
0929 20:32:11.767834 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:11.767854 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:11.768009 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:11.768876 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:11.768880 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:11.768972 (+ 92us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:32:11.769337 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:11.769350 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:11.770675 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:11.770684 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:11.773270 (+ 2586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:11.773290 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:11.775350 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:11.775357 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:11.775536 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:11.775571 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 20:32:11.776427 (+ 856us) client_negotiation.cc:240] Negotiation successful
0929 20:32:11.776585 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"mutex_wait_us":56,"spinlock_wait_cycles":26496,"thread_start_us":433,"threads_started":1}
I20260929 20:32:11.778138 10503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f8b6656824d444f8b04f6d0fed1dcf7" candidate_uuid: "9610f3ba6f514c359a49c3f5f477be22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f75ac368e2472fb436ceefe945e7d8" is_pre_election: true
I20260929 20:32:11.778139 10367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f8b6656824d444f8b04f6d0fed1dcf7" candidate_uuid: "9610f3ba6f514c359a49c3f5f477be22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d4ce3c6363b495d8e7f981a2cfb1a12" is_pre_election: true
I20260929 20:32:11.778826 10503 raft_consensus.cc:2454] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 85f75ac368e2472fb436ceefe945e7d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9610f3ba6f514c359a49c3f5f477be22 in term 0.
I20260929 20:32:11.778826 10367 raft_consensus.cc:2454] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9d4ce3c6363b495d8e7f981a2cfb1a12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9610f3ba6f514c359a49c3f5f477be22 in term 0.
I20260929 20:32:11.779589 10157 leader_election.cc:302] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9610f3ba6f514c359a49c3f5f477be22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f75ac368e2472fb436ceefe945e7d8, 9610f3ba6f514c359a49c3f5f477be22; no voters:
I20260929 20:32:11.780081 10584 raft_consensus.cc:2781] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9610f3ba6f514c359a49c3f5f477be22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:32:11.780263 10584 raft_consensus.cc:482] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9610f3ba6f514c359a49c3f5f477be22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:32:11.780382 10584 raft_consensus.cc:3037] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9610f3ba6f514c359a49c3f5f477be22 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:32:11.782963 10584 raft_consensus.cc:504] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9610f3ba6f514c359a49c3f5f477be22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d4ce3c6363b495d8e7f981a2cfb1a12" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33059 } } peers { permanent_uuid: "9610f3ba6f514c359a49c3f5f477be22" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 41821 } } peers { permanent_uuid: "85f75ac368e2472fb436ceefe945e7d8" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 42687 } }
I20260929 20:32:11.783756 10584 leader_election.cc:288] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9610f3ba6f514c359a49c3f5f477be22 [CANDIDATE]: Term 1 election: Requested vote from peers 9d4ce3c6363b495d8e7f981a2cfb1a12 (127.28.244.130:33059), 85f75ac368e2472fb436ceefe945e7d8 (127.28.244.131:42687)
I20260929 20:32:11.784328 10367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f8b6656824d444f8b04f6d0fed1dcf7" candidate_uuid: "9610f3ba6f514c359a49c3f5f477be22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d4ce3c6363b495d8e7f981a2cfb1a12"
I20260929 20:32:11.784598 10503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f8b6656824d444f8b04f6d0fed1dcf7" candidate_uuid: "9610f3ba6f514c359a49c3f5f477be22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f75ac368e2472fb436ceefe945e7d8"
I20260929 20:32:11.784631 10367 raft_consensus.cc:3037] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9d4ce3c6363b495d8e7f981a2cfb1a12 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:32:11.784852 10503 raft_consensus.cc:3037] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 85f75ac368e2472fb436ceefe945e7d8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:32:11.788465 10503 raft_consensus.cc:2454] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 85f75ac368e2472fb436ceefe945e7d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9610f3ba6f514c359a49c3f5f477be22 in term 1.
I20260929 20:32:11.788465 10367 raft_consensus.cc:2454] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9d4ce3c6363b495d8e7f981a2cfb1a12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9610f3ba6f514c359a49c3f5f477be22 in term 1.
I20260929 20:32:11.789023 10157 leader_election.cc:302] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9610f3ba6f514c359a49c3f5f477be22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f75ac368e2472fb436ceefe945e7d8, 9610f3ba6f514c359a49c3f5f477be22; no voters:
I20260929 20:32:11.789451 10584 raft_consensus.cc:2781] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9610f3ba6f514c359a49c3f5f477be22 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:32:11.790057 10584 raft_consensus.cc:686] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9610f3ba6f514c359a49c3f5f477be22 [term 1 LEADER]: Becoming Leader. State: Replica: 9610f3ba6f514c359a49c3f5f477be22, State: Running, Role: LEADER
I20260929 20:32:11.790741 10584 consensus_queue.cc:237] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9610f3ba6f514c359a49c3f5f477be22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d4ce3c6363b495d8e7f981a2cfb1a12" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33059 } } peers { permanent_uuid: "9610f3ba6f514c359a49c3f5f477be22" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 41821 } } peers { permanent_uuid: "85f75ac368e2472fb436ceefe945e7d8" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 42687 } }
I20260929 20:32:11.796784 10071 catalog_manager.cc:5495] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9610f3ba6f514c359a49c3f5f477be22 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9610f3ba6f514c359a49c3f5f477be22 (127.28.244.129). New cstate: current_term: 1 leader_uuid: "9610f3ba6f514c359a49c3f5f477be22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d4ce3c6363b495d8e7f981a2cfb1a12" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9610f3ba6f514c359a49c3f5f477be22" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 41821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85f75ac368e2472fb436ceefe945e7d8" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 42687 } health_report { overall_health: UNKNOWN } } }
I20260929 20:32:11.836717 10574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:11.829433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57572 (local address 127.28.244.129:41821)
0929 20:32:11.829651 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:11.829658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:11.829680 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:32:11.829878 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:11.829884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:11.829989 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:11.830167 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:32:11.830561 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:11.831627 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:11.833622 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:11.834431 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:11.834604 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:11.835264 (+ 660us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:32:11.836039 (+ 775us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:32:11.836212 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:11.836350 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 20:32:11.836444 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:32:11.846264 10503 raft_consensus.cc:1261] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 85f75ac368e2472fb436ceefe945e7d8 [term 1 FOLLOWER]: Refusing update from remote peer 9610f3ba6f514c359a49c3f5f477be22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:32:11.846323 10367 raft_consensus.cc:1261] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9d4ce3c6363b495d8e7f981a2cfb1a12 [term 1 FOLLOWER]: Refusing update from remote peer 9610f3ba6f514c359a49c3f5f477be22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:32:11.847393 10587 consensus_queue.cc:1035] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9610f3ba6f514c359a49c3f5f477be22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f75ac368e2472fb436ceefe945e7d8" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 42687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:32:11.847878 10584 consensus_queue.cc:1035] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9610f3ba6f514c359a49c3f5f477be22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d4ce3c6363b495d8e7f981a2cfb1a12" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:32:11.867431 10590 mvcc.cc:205] Tried to move back new op lower bound from 7334764264830410752 to 7334764264619945984. Current Snapshot: MvccSnapshot[applied={T|T < 7334764264830410752}]
I20260929 20:32:11.876058 10591 mvcc.cc:205] Tried to move back new op lower bound from 7334764264830410752 to 7334764264619945984. Current Snapshot: MvccSnapshot[applied={T|T < 7334764264830410752}]
I20260929 20:32:11.900743 10070 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:32:11.901120 10070 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:32:11.905290 10070 catalog_manager.cc:5782] T 00000000000000000000000000000000 P afea710921db4f3d8497301702112223: Sending DeleteTablet for 3 replicas of tablet 7f8b6656824d444f8b04f6d0fed1dcf7
I20260929 20:32:11.906976 10347 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f8b6656824d444f8b04f6d0fed1dcf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:32:11 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:56912
I20260929 20:32:11.907209 10482 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f8b6656824d444f8b04f6d0fed1dcf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:32:11 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:37786
I20260929 20:32:11.907212 10203 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f8b6656824d444f8b04f6d0fed1dcf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:32:11 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:57570
I20260929 20:32:11.913079 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 10108
I20260929 20:32:11.919516 10600 tablet_replica.cc:330] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 85f75ac368e2472fb436ceefe945e7d8: stopping tablet replica
I20260929 20:32:11.920847 10600 raft_consensus.cc:2229] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 85f75ac368e2472fb436ceefe945e7d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:32:11.922243 10600 raft_consensus.cc:2258] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 85f75ac368e2472fb436ceefe945e7d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:32:11.925812 10600 ts_tablet_manager.cc:1905] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 85f75ac368e2472fb436ceefe945e7d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:32:11.928414 10601 tablet_replica.cc:330] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9610f3ba6f514c359a49c3f5f477be22: stopping tablet replica
I20260929 20:32:11.929028 10601 raft_consensus.cc:2229] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9610f3ba6f514c359a49c3f5f477be22 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:32:11.929890 10601 raft_consensus.cc:2258] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9610f3ba6f514c359a49c3f5f477be22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:32:11.933751 10601 ts_tablet_manager.cc:1905] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9610f3ba6f514c359a49c3f5f477be22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:32:11.934250 10600 ts_tablet_manager.cc:1918] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 85f75ac368e2472fb436ceefe945e7d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:32:11.941414 10574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:11.914848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57586 (local address 127.28.244.129:41821)
0929 20:32:11.915050 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:11.915058 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:11.915078 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:32:11.915150 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:11.915155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:11.915240 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:11.915368 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:11.917811 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:11.919171 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:11.923018 (+ 3847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:11.923773 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:11.924466 (+ 693us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:11.928531 (+ 4065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:11.928548 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:11.928569 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:11.928635 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:11.933797 (+ 5162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:11.936087 (+ 2290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:11.936098 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:11.936102 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:11.936312 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:11.936855 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:11.936860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:11.936864 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:11.937367 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:11.940322 (+ 2955us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:11.940705 (+ 383us) server_negotiation.cc:299] Negotiation successful
0929 20:32:11.940986 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 20:32:11.942596 10263 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:32:11.934518 10600 log.cc:1196] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 85f75ac368e2472fb436ceefe945e7d8: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-2/wal/wals/7f8b6656824d444f8b04f6d0fed1dcf7
I20260929 20:32:11.944823 10600 ts_tablet_manager.cc:1939] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 85f75ac368e2472fb436ceefe945e7d8: Deleting consensus metadata
I20260929 20:32:11.945349 10599 tablet_replica.cc:330] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9d4ce3c6363b495d8e7f981a2cfb1a12: stopping tablet replica
I20260929 20:32:11.946035 10599 raft_consensus.cc:2229] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9d4ce3c6363b495d8e7f981a2cfb1a12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:32:11.947535 10599 raft_consensus.cc:2258] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9d4ce3c6363b495d8e7f981a2cfb1a12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:32:11.942255 10601 ts_tablet_manager.cc:1918] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9610f3ba6f514c359a49c3f5f477be22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:32:11.949153 10601 log.cc:1196] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9610f3ba6f514c359a49c3f5f477be22: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-0/wal/wals/7f8b6656824d444f8b04f6d0fed1dcf7
I20260929 20:32:11.949767 10599 ts_tablet_manager.cc:1905] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9d4ce3c6363b495d8e7f981a2cfb1a12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:32:11.950724 10057 catalog_manager.cc:4841] TS 85f75ac368e2472fb436ceefe945e7d8 (127.28.244.131:42687): tablet 7f8b6656824d444f8b04f6d0fed1dcf7 (table test-table [id=61f8dc4a251c4f0abde48ae68167f9a7]) successfully deleted
I20260929 20:32:11.957880 10599 ts_tablet_manager.cc:1918] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9d4ce3c6363b495d8e7f981a2cfb1a12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:32:11.958267 10599 log.cc:1196] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9d4ce3c6363b495d8e7f981a2cfb1a12: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713721201454-29650-0/minicluster-data/ts-1/wal/wals/7f8b6656824d444f8b04f6d0fed1dcf7
I20260929 20:32:11.959071 10599 ts_tablet_manager.cc:1939] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9d4ce3c6363b495d8e7f981a2cfb1a12: Deleting consensus metadata
I20260929 20:32:11.961390 10057 catalog_manager.cc:4841] TS 9d4ce3c6363b495d8e7f981a2cfb1a12 (127.28.244.130:33059): tablet 7f8b6656824d444f8b04f6d0fed1dcf7 (table test-table [id=61f8dc4a251c4f0abde48ae68167f9a7]) successfully deleted
I20260929 20:32:12.098693 10601 ts_tablet_manager.cc:1939] T 7f8b6656824d444f8b04f6d0fed1dcf7 P 9610f3ba6f514c359a49c3f5f477be22: Deleting consensus metadata
I20260929 20:32:12.100843 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 10108
I20260929 20:32:12.102792 10057 catalog_manager.cc:4841] TS 9610f3ba6f514c359a49c3f5f477be22 (127.28.244.129:41821): tablet 7f8b6656824d444f8b04f6d0fed1dcf7 (table test-table [id=61f8dc4a251c4f0abde48ae68167f9a7]) successfully deleted
I20260929 20:32:12.119951 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 10272
I20260929 20:32:12.135341 10572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:12.120751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56920 (local address 127.28.244.130:33059)
0929 20:32:12.120962 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:12.120967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:12.120988 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:32:12.121124 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:12.121128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:12.121210 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:12.121308 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:12.122757 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:12.123669 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:12.125115 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:12.125649 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:12.125770 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:12.128876 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:12.128903 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:12.128920 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:12.128964 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:12.132393 (+ 3429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:12.133350 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:12.133358 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:12.133360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:12.133503 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:12.133919 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:12.133923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:12.133925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:12.134323 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:12.134557 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:12.134886 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 20:32:12.135061 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 20:32:12.136166 10407 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:32:12.231614 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 10272
I20260929 20:32:12.249410 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 10416
I20260929 20:32:12.265196 10575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:12.250271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37788 (local address 127.28.244.131:42687)
0929 20:32:12.250464 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:12.250472 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:12.250493 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:32:12.250782 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:12.250786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:12.250868 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:12.250978 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:12.252796 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:12.253715 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:12.255205 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:12.255666 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:12.255978 (+ 312us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:12.258785 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:12.258803 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:12.258820 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:12.258860 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:12.261918 (+ 3058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:12.262745 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:12.262759 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:12.262767 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:12.262920 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:12.263375 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:12.263380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:12.263384 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:12.263798 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:12.264214 (+ 416us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:12.264590 (+ 376us) server_negotiation.cc:299] Negotiation successful
0929 20:32:12.264823 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:32:12.265974 10543 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:32:12.356940 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 10416
I20260929 20:32:12.374596 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 10038
I20260929 20:32:12.375998 10101 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:32:12.490760 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 10038
I20260929 20:32:12.514024 29650 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 20:32:12.860944 29650 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 20:32:12.861220 29650 mini_ranger.cc:66] Stopping Ranger...
I20260929 20:32:13.207978 29650 mini_ranger.cc:68] Stopped Ranger
2026-09-29 20:32:13.208 UTC [9090] LOG: received smart shutdown request
2026-09-29 20:32:13.211 UTC [9090] LOG: background worker "logical replication launcher" (PID 9099) exited with exit code 1
2026-09-29 20:32:13.211 UTC [9094] LOG: shutting down
2026-09-29 20:32:13.436 UTC [9090] LOG: database system is shut down
2026-09-29T20:32:13Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (56883 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:32:13 dist-test-slave-p29l krb5kdc[10614](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:32:13 dist-test-slave-p29l krb5kdc[10614](info): set up 2 sockets
Sep 29 20:32:13 dist-test-slave-p29l krb5kdc[10614](info): commencing operation
krb5kdc: starting...
W20260929 20:32:14.743850 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.137s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:32:14 dist-test-slave-p29l krb5kdc[10614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:32:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:32:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.244.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
I20260929 20:32:14.997888 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:34625
--webserver_interface=127.28.244.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:36589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:34625,127.28.244.189:45623,127.28.244.188:45429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:32:15.368466 10627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:32:15.368813 10627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:32:15.369005 10627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:32:15.376847 10627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:32:15.376973 10627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:32:15.377058 10627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:32:15.377157 10627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:32:15.377242 10627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:32:15.387856 10627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:36589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:34625,127.28.244.189:45623,127.28.244.188:45429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:34625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:32:15.389360 10627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:32:15.390795 10627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:32:15.404573 10633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:15.404667 10632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:15.404697 10635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:32:15.405083 10627 server_base.cc:1034] running on GCE node
I20260929 20:32:15.406140 10627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:32:15.407845 10627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:32:15.409114 10627 hybrid_clock.cc:648] HybridClock initialized: now 1790713935409053 us; error 57 us; skew 500 ppm
I20260929 20:32:15.409531 10627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:32:15 dist-test-slave-p29l krb5kdc[10614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713935, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:32:15.418509 10627 init.cc:377] Logged in from keytab as kudu/127.28.244.190@KRBTEST.COM (short username kudu)
I20260929 20:32:15.420341 10627 webserver.cc:460] Webserver started at http://127.28.244.190:38687/ using document root <none> and password file <none>
I20260929 20:32:15.421201 10627 fs_manager.cc:359] Metadata directory not provided
I20260929 20:32:15.421351 10627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:32:15.421710 10627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:32:15.425518 10627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "36378a5dbd1441d5970d36bf577de0bb"
format_stamp: "Formatted at 2026-09-29 20:32:15 on dist-test-slave-p29l"
server_key: "365645b20548df37cdec09eed48b21fc"
server_key_iv: "58c831cc4251783317099cac37d449c9"
server_key_version: "encryptionkey@0"
I20260929 20:32:15.426455 10627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "36378a5dbd1441d5970d36bf577de0bb"
format_stamp: "Formatted at 2026-09-29 20:32:15 on dist-test-slave-p29l"
server_key: "365645b20548df37cdec09eed48b21fc"
server_key_iv: "58c831cc4251783317099cac37d449c9"
server_key_version: "encryptionkey@0"
I20260929 20:32:15.432441 10627 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 20:32:15.436653 10642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:15.438359 10627 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 20:32:15.438599 10627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "36378a5dbd1441d5970d36bf577de0bb"
format_stamp: "Formatted at 2026-09-29 20:32:15 on dist-test-slave-p29l"
server_key: "365645b20548df37cdec09eed48b21fc"
server_key_iv: "58c831cc4251783317099cac37d449c9"
server_key_version: "encryptionkey@0"
I20260929 20:32:15.438843 10627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:32:15.459304 10627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:32:15.464005 10627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:32:15.464319 10627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:32:15.484794 10627 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:34625
I20260929 20:32:15.484830 10694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:34625 every 8 connection(s)
I20260929 20:32:15.486325 10627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:32:15.490466 10695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:32:15.494514 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 10627
I20260929 20:32:15.494803 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:32:15.495461 29650 external_mini_cluster.cc:1428] Setting key 1c7c6f982f62f51de7c623c4fea10bd6
I20260929 20:32:15.497980 10695 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34625 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:32:15 dist-test-slave-p29l krb5kdc[10614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713935, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.244.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.189@KRBTEST.COM" created.
I20260929 20:32:15.528378 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:15.501856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:34625 (local address 127.0.0.1:42974)
0929 20:32:15.503875 (+ 2019us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:15.503932 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:15.507384 (+ 3452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:15.508700 (+ 1316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:15.508711 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:15.509613 (+ 902us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:32:15.510210 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:15.510244 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:15.512008 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:15.512016 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:15.514121 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:15.514142 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:15.515481 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:15.515493 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:15.515760 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:15.516574 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:32:15.516617 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:32:15.519631 (+ 3014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:32:15.523801 (+ 4170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:15.523862 (+ 61us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:15.523880 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:15.524354 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:15.524877 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:15.524880 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:15.524882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:15.525044 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:15.526184 (+ 1140us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:32:15.526191 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:32:15.526429 (+ 238us) client_negotiation.cc:769] Sending connection context
0929 20:32:15.526722 (+ 293us) client_negotiation.cc:240] Negotiation successful
0929 20:32:15.527166 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1677,"thread_start_us":1383,"threads_started":1}
I20260929 20:32:15.528436 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:15.501855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42974 (local address 127.28.244.190:34625)
0929 20:32:15.502916 (+ 1061us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:15.502937 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:15.504114 (+ 1177us) server_negotiation.cc:407] Connection header received
0929 20:32:15.508059 (+ 3945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:15.508067 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:15.508392 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:15.508582 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:15.510464 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:15.511762 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:15.514322 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:15.515189 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:15.515355 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:15.519938 (+ 4583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:15.519948 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:15.519962 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:15.520003 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:15.523606 (+ 3603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:15.524511 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:15.524523 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:15.524529 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:15.524744 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:15.525166 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:15.525174 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:15.525179 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:15.526012 (+ 833us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:15.526239 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:15.526933 (+ 694us) server_negotiation.cc:299] Negotiation successful
0929 20:32:15.527391 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":733,"thread_start_us":1011,"threads_started":1}
I20260929 20:32:15.531601 10695 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.244.189" port: 45623 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:32:15.532924 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:15.532323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:45623 (local address 127.0.0.1:40018)
0929 20:32:15.532553 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:15.532702 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.189:45623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260929 20:32:15.533232 10643 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.244.189:45623: connect: Connection refused (error 111)
I20260929 20:32:15.535104 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:15.534594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:45623 (local address 127.0.0.1:40034)
0929 20:32:15.534797 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:15.534901 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.189:45623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 20:32:15.535333 10695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.189:45623: Network error: Client connection negotiation failed: client connection to 127.28.244.189:45623: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.189.keytab.
Entry for principal kudu/127.28.244.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.189.keytab.
I20260929 20:32:15.555254 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.189:45623
--webserver_interface=127.28.244.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.189.keytab
--principal=kudu/127.28.244.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:36589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:34625,127.28.244.189:45623,127.28.244.188:45429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:32:15.584759 10695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.189" port: 45623 } attempt: 1
I20260929 20:32:15.586092 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:15.585529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:45623 (local address 127.0.0.1:40038)
0929 20:32:15.585774 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:15.585911 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.189:45623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260929 20:32:15.587291 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:15.586885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:45623 (local address 127.0.0.1:40050)
0929 20:32:15.587068 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:15.587144 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.189:45623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 20:32:15.587556 10695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.189:45623: Network error: Client connection negotiation failed: client connection to 127.28.244.189:45623: connect: Connection refused (error 111)
I20260929 20:32:15.655920 10695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.189" port: 45623 } attempt: 2
I20260929 20:32:15.657316 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:15.656766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:45623 (local address 127.0.0.1:40052)
0929 20:32:15.657054 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:15.657157 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.189:45623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 20:32:15.658948 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:15.658277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:45623 (local address 127.0.0.1:40056)
0929 20:32:15.658481 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:15.658582 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.189:45623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260929 20:32:15.659304 10695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.189:45623: Network error: Client connection negotiation failed: client connection to 127.28.244.189:45623: connect: Connection refused (error 111)
I20260929 20:32:15.750624 10695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.189" port: 45623 } attempt: 3
I20260929 20:32:15.751899 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:15.751380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:45623 (local address 127.0.0.1:40068)
0929 20:32:15.751590 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:15.751697 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.189:45623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 20:32:15.752998 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:15.752579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:45623 (local address 127.0.0.1:40082)
0929 20:32:15.752768 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:15.752858 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.189:45623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 20:32:15.753221 10695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.189:45623: Network error: Client connection negotiation failed: client connection to 127.28.244.189:45623: connect: Connection refused (error 111)
I20260929 20:32:15.896482 10695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.189" port: 45623 } attempt: 4
I20260929 20:32:15.897778 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:15.897239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:45623 (local address 127.0.0.1:40092)
0929 20:32:15.897539 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:15.897626 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.189:45623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260929 20:32:15.898885 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:15.898519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:45623 (local address 127.0.0.1:40100)
0929 20:32:15.898671 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:15.898765 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.189:45623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 20:32:15.899057 10695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.189:45623: Network error: Client connection negotiation failed: client connection to 127.28.244.189:45623: connect: Connection refused (error 111)
W20260929 20:32:15.914114 10702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:32:15.914498 10702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:32:15.914659 10702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:32:15.922178 10702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:32:15.922344 10702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:32:15.922449 10702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:32:15.922545 10702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:32:15.922617 10702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:32:15.933256 10702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:36589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:34625,127.28.244.189:45623,127.28.244.188:45429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.189.keytab
--principal=kudu/127.28.244.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.189:45623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.244.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:32:15.934832 10702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:32:15.936316 10702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:32:15.949501 10708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:15.949455 10707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:15.949422 10710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:32:15.950600 10702 server_base.cc:1034] running on GCE node
I20260929 20:32:15.951514 10702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:32:15.953171 10702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:32:15.954422 10702 hybrid_clock.cc:648] HybridClock initialized: now 1790713935954365 us; error 53 us; skew 500 ppm
I20260929 20:32:15.954857 10702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:32:15 dist-test-slave-p29l krb5kdc[10614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713935, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:32:15.958534 10702 init.cc:377] Logged in from keytab as kudu/127.28.244.189@KRBTEST.COM (short username kudu)
I20260929 20:32:15.960214 10702 webserver.cc:460] Webserver started at http://127.28.244.189:34541/ using document root <none> and password file <none>
I20260929 20:32:15.960928 10702 fs_manager.cc:359] Metadata directory not provided
I20260929 20:32:15.961035 10702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:32:15.961306 10702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:32:15.964583 10702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-1/data/instance:
uuid: "14d03573df38454ba0d41fdf35c31205"
format_stamp: "Formatted at 2026-09-29 20:32:15 on dist-test-slave-p29l"
server_key: "22ca47e008e3c4e31e93b8145af62378"
server_key_iv: "afce70c57d7bba0d2b1ca6d808931e75"
server_key_version: "encryptionkey@0"
I20260929 20:32:15.965423 10702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-1/wal/instance:
uuid: "14d03573df38454ba0d41fdf35c31205"
format_stamp: "Formatted at 2026-09-29 20:32:15 on dist-test-slave-p29l"
server_key: "22ca47e008e3c4e31e93b8145af62378"
server_key_iv: "afce70c57d7bba0d2b1ca6d808931e75"
server_key_version: "encryptionkey@0"
I20260929 20:32:15.971093 10702 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:32:15.975025 10717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:15.976747 10702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:32:15.976927 10702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-1/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-1/wal
uuid: "14d03573df38454ba0d41fdf35c31205"
format_stamp: "Formatted at 2026-09-29 20:32:15 on dist-test-slave-p29l"
server_key: "22ca47e008e3c4e31e93b8145af62378"
server_key_iv: "afce70c57d7bba0d2b1ca6d808931e75"
server_key_version: "encryptionkey@0"
I20260929 20:32:15.977129 10702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:32:16.008136 10702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:32:16.013469 10702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:32:16.013756 10702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:32:16.033533 10702 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.189:45623
I20260929 20:32:16.033555 10768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.189:45623 every 8 connection(s)
I20260929 20:32:16.035068 10702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-1/data/info.pb
I20260929 20:32:16.039299 10769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:32:16.040185 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 10702
I20260929 20:32:16.040386 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-1/wal/instance
I20260929 20:32:16.040951 29650 external_mini_cluster.cc:1428] Setting key 08e06dca22c9eec934b9923e70dc0952
I20260929 20:32:16.050729 10769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34625 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.244.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.188@KRBTEST.COM" created.
Sep 29 20:32:16 dist-test-slave-p29l krb5kdc[10614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713935, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.189@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:32:16.080554 10772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:16.053829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42990 (local address 127.28.244.190:34625)
0929 20:32:16.054444 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:16.054452 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:16.055625 (+ 1173us) server_negotiation.cc:407] Connection header received
0929 20:32:16.059072 (+ 3447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:16.059080 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:16.059188 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:16.059306 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:16.062184 (+ 2878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.063428 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.066411 (+ 2983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.066878 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.067011 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:16.071598 (+ 4587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:16.071608 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:16.071616 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:16.071686 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:16.075939 (+ 4253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:16.077429 (+ 1490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:16.077436 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:16.077444 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:16.077598 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:16.078403 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:16.078409 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:16.078411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:16.078735 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:16.078876 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:16.080030 (+ 1154us) server_negotiation.cc:299] Negotiation successful
0929 20:32:16.080266 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":306,"threads_started":1}
I20260929 20:32:16.082500 10773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:16.054459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:34625 (local address 127.0.0.1:42990)
0929 20:32:16.055393 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:16.055461 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:16.058583 (+ 3122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:16.059699 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:16.059718 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:16.060725 (+ 1007us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:32:16.061762 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:16.061807 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.063715 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.063722 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:16.066122 (+ 2400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:16.066146 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.067222 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.067229 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:16.067477 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:16.068290 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:32:16.068316 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:32:16.071242 (+ 2926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:32:16.076216 (+ 4974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:16.076227 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:16.076245 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:16.077203 (+ 958us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:16.077754 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:16.077761 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:16.077768 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:16.078232 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:16.078898 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:32:16.078912 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:32:16.079473 (+ 561us) client_negotiation.cc:769] Sending connection context
0929 20:32:16.079896 (+ 423us) client_negotiation.cc:240] Negotiation successful
0929 20:32:16.080698 (+ 802us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":341,"threads_started":1}
I20260929 20:32:16.084970 10769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.244.189" port: 45623 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:32:16 dist-test-slave-p29l krb5kdc[10614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713935, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.189@KRBTEST.COM for kudu/127.28.244.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.188.keytab.
Entry for principal kudu/127.28.244.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.188.keytab.
I20260929 20:32:16.105482 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.188:45429
--webserver_interface=127.28.244.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.188.keytab
--principal=kudu/127.28.244.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:36589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:34625,127.28.244.189:45623,127.28.244.188:45429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:32:16.106189 10773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:16.086039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:45623 (local address 127.0.0.1:40106)
0929 20:32:16.086333 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:16.086357 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:16.086598 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:16.088725 (+ 2127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:16.088733 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:16.089330 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:32:16.089576 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:16.089596 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.091111 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.091119 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:16.092716 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:16.092737 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.093833 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.093846 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:16.094087 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:16.094891 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:32:16.094922 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:32:16.097959 (+ 3037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:32:16.103096 (+ 5137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:16.103101 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:16.103104 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:16.103582 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:16.104233 (+ 651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:16.104241 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:16.104245 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:16.104398 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:16.105252 (+ 854us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:32:16.105257 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:32:16.105393 (+ 136us) client_negotiation.cc:769] Sending connection context
0929 20:32:16.105564 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 20:32:16.105926 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260929 20:32:16.106758 10775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:16.086809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40106 (local address 127.28.244.189:45623)
0929 20:32:16.088153 (+ 1344us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:16.088160 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:16.088196 (+ 36us) server_negotiation.cc:407] Connection header received
0929 20:32:16.088264 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:16.088271 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:16.088504 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:16.088658 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:16.089772 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.090935 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.092929 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.093593 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.093768 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:16.098335 (+ 4567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:16.098356 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:16.098374 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:16.098417 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:16.102741 (+ 4324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:16.103862 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:16.103879 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:16.103882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:16.104086 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:16.104517 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:16.104522 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:16.104525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:16.105089 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:16.105381 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:16.105754 (+ 373us) server_negotiation.cc:299] Negotiation successful
0929 20:32:16.106156 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":880,"thread_start_us":668,"threads_started":1}
I20260929 20:32:16.109493 10769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.244.188" port: 45429 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:32:16.110912 10773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:16.110205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:45429 (local address 127.0.0.1:35376)
0929 20:32:16.110480 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:16.110682 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260929 20:32:16.111263 10719 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.244.188:45429: connect: Connection refused (error 111)
I20260929 20:32:16.113255 10773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:16.112804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:45429 (local address 127.0.0.1:35386)
0929 20:32:16.113022 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:16.113138 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 20:32:16.113517 10769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.188:45429: Network error: Client connection negotiation failed: client connection to 127.28.244.188:45429: connect: Connection refused (error 111)
I20260929 20:32:16.162868 10769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.188" port: 45429 } attempt: 1
I20260929 20:32:16.164445 10773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:16.163709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:45429 (local address 127.0.0.1:35388)
0929 20:32:16.163997 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:16.164268 (+ 271us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260929 20:32:16.165650 10773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:16.165178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:45429 (local address 127.0.0.1:35394)
0929 20:32:16.165420 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:16.165517 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 20:32:16.165913 10769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.188:45429: Network error: Client connection negotiation failed: client connection to 127.28.244.188:45429: connect: Connection refused (error 111)
I20260929 20:32:16.198336 10695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.189" port: 45623 } attempt: 5
Sep 29 20:32:16 dist-test-slave-p29l krb5kdc[10614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713935, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for kudu/127.28.244.189@KRBTEST.COM
I20260929 20:32:16.213335 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:16.199345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:45623 (local address 127.0.0.1:40114)
0929 20:32:16.199613 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:16.199637 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:16.199796 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:16.200249 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:16.200253 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:16.200744 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:32:16.200938 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:16.200949 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.202079 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.202085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:16.203010 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:16.203024 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.203878 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.203884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:16.204085 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:16.204822 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:32:16.204842 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:32:16.207053 (+ 2211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:32:16.210528 (+ 3475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:16.210535 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:16.210539 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:16.210970 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:16.211453 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:16.211458 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:16.211462 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:16.211617 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:16.212245 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:32:16.212250 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:32:16.212425 (+ 175us) client_negotiation.cc:769] Sending connection context
0929 20:32:16.212597 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 20:32:16.212971 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 20:32:16.213333 10775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:16.199452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40114 (local address 127.28.244.189:45623)
0929 20:32:16.199658 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:16.199666 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:16.199714 (+ 48us) server_negotiation.cc:407] Connection header received
0929 20:32:16.200012 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:16.200017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:16.200100 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:16.200208 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:16.201214 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.201925 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.203193 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.203679 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.203856 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:16.207301 (+ 3445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:16.207307 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:16.207310 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:16.207347 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:16.210325 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:16.211113 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:16.211119 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:16.211122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:16.211315 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:16.211736 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:16.211742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:16.211745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:16.212115 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:16.212296 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:16.212728 (+ 432us) server_negotiation.cc:299] Negotiation successful
0929 20:32:16.212973 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 20:32:16.214973 10695 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.244.188" port: 45429 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:32:16.216384 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:16.215774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:45429 (local address 127.0.0.1:35398)
0929 20:32:16.216059 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:16.216172 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260929 20:32:16.217568 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:16.217114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:45429 (local address 127.0.0.1:35400)
0929 20:32:16.217274 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:16.217382 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 20:32:16.217820 10695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.188:45429: Network error: Client connection negotiation failed: client connection to 127.28.244.188:45429: connect: Connection refused (error 111)
I20260929 20:32:16.234138 10769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.188" port: 45429 } attempt: 2
I20260929 20:32:16.235610 10773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:16.234930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:45429 (local address 127.0.0.1:35414)
0929 20:32:16.235290 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:16.235405 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260929 20:32:16.236750 10773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:16.236396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:45429 (local address 127.0.0.1:35430)
0929 20:32:16.236550 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:16.236632 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 20:32:16.236995 10769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.188:45429: Network error: Client connection negotiation failed: client connection to 127.28.244.188:45429: connect: Connection refused (error 111)
I20260929 20:32:16.269110 10695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.188" port: 45429 } attempt: 1
I20260929 20:32:16.270701 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:16.269909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:45429 (local address 127.0.0.1:35446)
0929 20:32:16.270347 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:16.270479 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260929 20:32:16.271979 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:16.271521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:45429 (local address 127.0.0.1:35452)
0929 20:32:16.271699 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:16.271794 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260929 20:32:16.272223 10695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.188:45429: Network error: Client connection negotiation failed: client connection to 127.28.244.188:45429: connect: Connection refused (error 111)
I20260929 20:32:16.328225 10769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.188" port: 45429 } attempt: 3
I20260929 20:32:16.329432 10773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:16.328961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:45429 (local address 127.0.0.1:35458)
0929 20:32:16.329158 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:16.329262 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 20:32:16.330689 10773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:16.330184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:45429 (local address 127.0.0.1:35470)
0929 20:32:16.330430 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:16.330513 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
W20260929 20:32:16.331005 10769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.188:45429: Network error: Client connection negotiation failed: client connection to 127.28.244.188:45429: connect: Connection refused (error 111)
I20260929 20:32:16.340490 10695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.188" port: 45429 } attempt: 2
I20260929 20:32:16.341850 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:16.341292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:45429 (local address 127.0.0.1:35474)
0929 20:32:16.341546 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:16.341664 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 20:32:16.343017 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:16.342612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:45429 (local address 127.0.0.1:35484)
0929 20:32:16.342800 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:16.342891 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 20:32:16.343269 10695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.188:45429: Network error: Client connection negotiation failed: client connection to 127.28.244.188:45429: connect: Connection refused (error 111)
I20260929 20:32:16.449536 10695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.188" port: 45429 } attempt: 3
I20260929 20:32:16.450809 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:16.450319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:45429 (local address 127.0.0.1:35492)
0929 20:32:16.450539 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:16.450641 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 20:32:16.452246 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:16.451723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:45429 (local address 127.0.0.1:35496)
0929 20:32:16.451993 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:16.452083 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
W20260929 20:32:16.452461 10695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.188:45429: Network error: Client connection negotiation failed: client connection to 127.28.244.188:45429: connect: Connection refused (error 111)
I20260929 20:32:16.474283 10769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.188" port: 45429 } attempt: 4
I20260929 20:32:16.475729 10773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:16.475189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:45429 (local address 127.0.0.1:35508)
0929 20:32:16.475413 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:16.475517 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260929 20:32:16.476962 10773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:16.476506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:45429 (local address 127.0.0.1:35518)
0929 20:32:16.476730 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:16.476823 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260929 20:32:16.477226 10769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.188:45429: Network error: Client connection negotiation failed: client connection to 127.28.244.188:45429: connect: Connection refused (error 111)
W20260929 20:32:16.486294 10777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:32:16.486658 10777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:32:16.486809 10777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:32:16.494781 10777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:32:16.494951 10777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:32:16.495061 10777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:32:16.495162 10777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:32:16.495235 10777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:32:16.506034 10777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:36589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:34625,127.28.244.189:45623,127.28.244.188:45429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.188.keytab
--principal=kudu/127.28.244.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.188:45429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.244.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:32:16.507529 10777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:32:16.508976 10777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:32:16.522536 10782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:16.522472 10785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:16.522441 10783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:32:16.523483 10777 server_base.cc:1034] running on GCE node
I20260929 20:32:16.524483 10777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:32:16.526181 10777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:32:16.527443 10777 hybrid_clock.cc:648] HybridClock initialized: now 1790713936527388 us; error 53 us; skew 500 ppm
I20260929 20:32:16.527948 10777 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:32:16 dist-test-slave-p29l krb5kdc[10614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713936, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:32:16.531710 10777 init.cc:377] Logged in from keytab as kudu/127.28.244.188@KRBTEST.COM (short username kudu)
I20260929 20:32:16.533565 10777 webserver.cc:460] Webserver started at http://127.28.244.188:35239/ using document root <none> and password file <none>
I20260929 20:32:16.534403 10777 fs_manager.cc:359] Metadata directory not provided
I20260929 20:32:16.534548 10777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:32:16.534909 10777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:32:16.538401 10777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-2/data/instance:
uuid: "77687716a2c945fa9e927cfcd7e50641"
format_stamp: "Formatted at 2026-09-29 20:32:16 on dist-test-slave-p29l"
server_key: "99ee3bf985b0e06492889125d9cdf7d0"
server_key_iv: "8aca0500030cd23998f543ff1bfc384c"
server_key_version: "encryptionkey@0"
I20260929 20:32:16.539354 10777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-2/wal/instance:
uuid: "77687716a2c945fa9e927cfcd7e50641"
format_stamp: "Formatted at 2026-09-29 20:32:16 on dist-test-slave-p29l"
server_key: "99ee3bf985b0e06492889125d9cdf7d0"
server_key_iv: "8aca0500030cd23998f543ff1bfc384c"
server_key_version: "encryptionkey@0"
I20260929 20:32:16.545552 10777 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260929 20:32:16.549741 10793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:16.551409 10777 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:32:16.551659 10777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-2/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-2/wal
uuid: "77687716a2c945fa9e927cfcd7e50641"
format_stamp: "Formatted at 2026-09-29 20:32:16 on dist-test-slave-p29l"
server_key: "99ee3bf985b0e06492889125d9cdf7d0"
server_key_iv: "8aca0500030cd23998f543ff1bfc384c"
server_key_version: "encryptionkey@0"
I20260929 20:32:16.551956 10777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:32:16.586084 10777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:32:16.591095 10777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:32:16.591426 10777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:32:16.611392 10777 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.188:45429
I20260929 20:32:16.611434 10844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.188:45429 every 8 connection(s)
I20260929 20:32:16.612972 10777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-2/data/info.pb
I20260929 20:32:16.613517 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 10777
I20260929 20:32:16.613770 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713721201454-29650-0/minicluster-data/master-2/wal/instance
I20260929 20:32:16.614427 29650 external_mini_cluster.cc:1428] Setting key b3c411d3af9aca4eb8a2bb0ff3e7ddfa
I20260929 20:32:16.617923 10845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:32:16.627456 10845 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34625 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:32:16.629690 10695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.188" port: 45429 } attempt: 4
Sep 29 20:32:16 dist-test-slave-p29l krb5kdc[10614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.188@KRBTEST.COM
Sep 29 20:32:16 dist-test-slave-p29l krb5kdc[10614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713935, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for kudu/127.28.244.188@KRBTEST.COM
Sep 29 20:32:16 dist-test-slave-p29l krb5kdc[10614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713936, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.188@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:32:16.645392 10848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:16.616554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35530 (local address 127.28.244.188:45429)
0929 20:32:16.617409 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:16.617428 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:16.617484 (+ 56us) server_negotiation.cc:407] Connection header received
0929 20:32:16.619896 (+ 2412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:16.619930 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:16.620475 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:16.620972 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:16.622504 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.624703 (+ 2199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.626211 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.627339 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.627552 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:16.632958 (+ 5406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:16.632971 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:16.632988 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:16.633041 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:16.638555 (+ 5514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:16.639704 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:16.639716 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:16.639730 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:16.640018 (+ 288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:16.640550 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:16.640556 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:16.640567 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:16.641373 (+ 806us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:16.641701 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:16.643053 (+ 1352us) server_negotiation.cc:299] Negotiation successful
0929 20:32:16.643607 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"spinlock_wait_cycles":34176,"thread_start_us":336,"threads_started":1}
I20260929 20:32:16.649372 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:16.630429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:45429 (local address 127.0.0.1:35540)
0929 20:32:16.630679 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:16.630708 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:16.630843 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:16.633092 (+ 2249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:16.633099 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:16.633615 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:32:16.633832 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:16.633846 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.635476 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.635481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:16.637015 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:16.637035 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.638086 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.638099 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:16.638391 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:16.639073 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:32:16.639092 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:32:16.641865 (+ 2773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:32:16.646145 (+ 4280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:16.646151 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:16.646155 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:16.646694 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:16.647188 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:16.647195 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:16.647198 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:16.647371 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:16.648155 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:32:16.648160 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:32:16.648355 (+ 195us) client_negotiation.cc:769] Sending connection context
0929 20:32:16.648603 (+ 248us) client_negotiation.cc:240] Negotiation successful
0929 20:32:16.648931 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":34}
I20260929 20:32:16.649819 10851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:16.632158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35540 (local address 127.28.244.188:45429)
0929 20:32:16.632759 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:16.632765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:16.632785 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:32:16.632826 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:16.632830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:16.632911 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:16.633025 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:16.634004 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.635320 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.637204 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.637892 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.638035 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:16.642204 (+ 4169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:16.642211 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:16.642215 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:16.642266 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:16.645971 (+ 3705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:16.646855 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:16.646859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:16.646862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:16.647044 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:16.647493 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:16.647497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:16.647500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:16.648009 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:16.648175 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:16.648764 (+ 589us) server_negotiation.cc:299] Negotiation successful
0929 20:32:16.649023 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"spinlock_wait_cycles":31104,"thread_start_us":302,"threads_started":1}
I20260929 20:32:16.651862 10849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:16.630719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43000 (local address 127.28.244.190:34625)
0929 20:32:16.632305 (+ 1586us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:16.632311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:16.632333 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:32:16.632396 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:16.632402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:16.632497 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:16.632627 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:16.634401 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.635466 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.637631 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.638397 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.638537 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:16.643251 (+ 4714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:16.643258 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:16.643262 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:16.643311 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:16.648060 (+ 4749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:16.649471 (+ 1411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:16.649476 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:16.649479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:16.649612 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:16.650066 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:16.650070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:16.650073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:16.650396 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:16.650509 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:16.651196 (+ 687us) server_negotiation.cc:299] Negotiation successful
0929 20:32:16.651381 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":1454,"thread_start_us":1350,"threads_started":1}
I20260929 20:32:16.652268 10850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:16.630557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:34625 (local address 127.0.0.1:43000)
0929 20:32:16.631600 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:16.631662 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:16.631854 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:16.632869 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:16.632875 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:16.633838 (+ 963us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:32:16.634236 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:16.634247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.635659 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.635670 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:16.637469 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:16.637485 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.638621 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.638631 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:16.639191 (+ 560us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:16.639922 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:32:16.639949 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:32:16.642969 (+ 3020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:32:16.648674 (+ 5705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:16.648685 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:16.648694 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:16.649302 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:16.649735 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:16.649741 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:16.649744 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:16.649919 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:16.650542 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:32:16.650555 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:32:16.650840 (+ 285us) client_negotiation.cc:769] Sending connection context
0929 20:32:16.651100 (+ 260us) client_negotiation.cc:240] Negotiation successful
0929 20:32:16.651625 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"thread_start_us":560,"threads_started":1}
I20260929 20:32:16.657609 10845 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.244.189" port: 45623 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:32:16.662667 10695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36378a5dbd1441d5970d36bf577de0bb: Bootstrap starting.
Sep 29 20:32:16 dist-test-slave-p29l krb5kdc[10614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:32:16.667439 10695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36378a5dbd1441d5970d36bf577de0bb: Neither blocks nor log segments found. Creating new log.
Sep 29 20:32:16 dist-test-slave-p29l krb5kdc[10614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713936, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.188@KRBTEST.COM for kudu/127.28.244.189@KRBTEST.COM
I20260929 20:32:16.669235 10695 log.cc:824] T 00000000000000000000000000000000 P 36378a5dbd1441d5970d36bf577de0bb: Log is configured to *not* fsync() on all Append() calls
I20260929 20:32:16.673164 10849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:16.649530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43006 (local address 127.28.244.190:34625)
0929 20:32:16.652137 (+ 2607us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:16.652142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:16.652998 (+ 856us) server_negotiation.cc:407] Connection header received
0929 20:32:16.653119 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:16.653123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:16.653191 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:16.653278 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:16.655509 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.656559 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.658695 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.659356 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.659493 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:16.665567 (+ 6074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:16.665572 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:16.665576 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:16.665613 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:16.669517 (+ 3904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:16.670539 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:16.670548 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:16.670554 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:16.670736 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:16.671277 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:16.671283 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:16.671287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:16.671625 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:16.671979 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:16.672323 (+ 344us) server_negotiation.cc:299] Negotiation successful
0929 20:32:16.672729 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2488,"thread_start_us":6528,"threads_started":1}
I20260929 20:32:16.673916 10695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36378a5dbd1441d5970d36bf577de0bb: No bootstrap required, opened a new log
I20260929 20:32:16.676402 10850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:16.658367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:45623 (local address 127.0.0.1:40130)
0929 20:32:16.658560 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:16.658581 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:16.658685 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:16.660404 (+ 1719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:16.660410 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:16.660816 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:32:16.660974 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:16.660982 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.662455 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.662462 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:16.663851 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:16.663871 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.664979 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.664986 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:16.665248 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:16.666077 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:32:16.666101 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:32:16.668676 (+ 2575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:32:16.673203 (+ 4527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:16.673210 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:16.673216 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:16.673741 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:16.674251 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:16.674260 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:16.674267 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:16.674487 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:16.675266 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:32:16.675271 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:32:16.675464 (+ 193us) client_negotiation.cc:769] Sending connection context
0929 20:32:16.675623 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 20:32:16.675975 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 20:32:16.677255 10775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:16.659866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40130 (local address 127.28.244.189:45623)
0929 20:32:16.660104 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:16.660110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:16.660133 (+ 23us) server_negotiation.cc:407] Connection header received
0929 20:32:16.660187 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:16.660191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:16.660250 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:16.660368 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:16.661137 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.662248 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.664035 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.664757 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.664905 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:16.668940 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:16.668946 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:16.668949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:16.668990 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:16.673014 (+ 4024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:16.673886 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:16.673892 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:16.673895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:16.674088 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:16.674676 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:16.674682 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:16.674685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:16.675127 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:16.675303 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:16.676794 (+ 1491us) server_negotiation.cc:299] Negotiation successful
0929 20:32:16.676997 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 20:32:16.678128 10845 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.244.188" port: 45429 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:32:16.686152 10695 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36378a5dbd1441d5970d36bf577de0bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36378a5dbd1441d5970d36bf577de0bb" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34625 } } peers { permanent_uuid: "14d03573df38454ba0d41fdf35c31205" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 45623 } } peers { permanent_uuid: "77687716a2c945fa9e927cfcd7e50641" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 45429 } }
I20260929 20:32:16.686672 10695 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36378a5dbd1441d5970d36bf577de0bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:32:16.686810 10695 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36378a5dbd1441d5970d36bf577de0bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36378a5dbd1441d5970d36bf577de0bb, State: Initialized, Role: FOLLOWER
Sep 29 20:32:16 dist-test-slave-p29l krb5kdc[10614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713936, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.188@KRBTEST.COM for kudu/127.28.244.188@KRBTEST.COM
I20260929 20:32:16.687638 10695 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36378a5dbd1441d5970d36bf577de0bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36378a5dbd1441d5970d36bf577de0bb" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34625 } } peers { permanent_uuid: "14d03573df38454ba0d41fdf35c31205" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 45623 } } peers { permanent_uuid: "77687716a2c945fa9e927cfcd7e50641" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 45429 } }
I20260929 20:32:16.688773 10855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36378a5dbd1441d5970d36bf577de0bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36378a5dbd1441d5970d36bf577de0bb" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34625 } } peers { permanent_uuid: "14d03573df38454ba0d41fdf35c31205" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 45623 } } peers { permanent_uuid: "77687716a2c945fa9e927cfcd7e50641" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 45429 } } }
Sep 29 20:32:16 dist-test-slave-p29l krb5kdc[10614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.189@KRBTEST.COM
I20260929 20:32:16.690589 10855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36378a5dbd1441d5970d36bf577de0bb [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:32:16.690590 10695 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36378a5dbd1441d5970d36bf577de0bb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:32:16.695261 10851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:16.679210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35544 (local address 127.28.244.188:45429)
0929 20:32:16.679397 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:16.679404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:16.679425 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:32:16.679478 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:16.679485 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:16.679563 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:16.679673 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:16.680681 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.681582 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.683340 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.684176 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.684328 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:16.688481 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:16.688490 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:16.688496 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:16.688539 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:16.692058 (+ 3519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:16.693021 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:16.693030 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:16.693034 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:16.693213 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:16.693686 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:16.693692 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:16.693696 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:16.694075 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:16.694173 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:16.694603 (+ 430us) server_negotiation.cc:299] Negotiation successful
0929 20:32:16.694961 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":146,"server-negotiator.queue_time_us":40}
I20260929 20:32:16.695281 10850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:16.679004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:45429 (local address 127.0.0.1:35544)
0929 20:32:16.679247 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:16.679268 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:16.679372 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:16.679736 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:16.679740 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:16.680323 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:32:16.680484 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:16.680493 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.681748 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.681752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:16.683165 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:16.683181 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.684362 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.684367 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:16.684524 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:16.685237 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:32:16.685255 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:32:16.688245 (+ 2990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:32:16.692256 (+ 4011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:16.692265 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:16.692270 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:16.692878 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:16.693368 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:16.693375 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:16.693379 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:16.693545 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:16.694230 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:32:16.694234 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:32:16.694344 (+ 110us) client_negotiation.cc:769] Sending connection context
0929 20:32:16.694487 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 20:32:16.694825 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 20:32:16.700584 10854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:16.675878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40138 (local address 127.28.244.189:45623)
0929 20:32:16.676522 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:16.676528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:16.676555 (+ 27us) server_negotiation.cc:407] Connection header received
0929 20:32:16.676612 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:16.676618 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:16.676699 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:16.679838 (+ 3139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:16.680116 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.681525 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.684559 (+ 3034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.685299 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.685441 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:16.691270 (+ 5829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:16.691281 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:16.691286 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:16.691336 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:16.695980 (+ 4644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:16.697046 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:16.697053 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:16.697057 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:16.697238 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:16.697798 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:16.697804 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:16.697807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:16.698210 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:16.698474 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:16.699893 (+ 1419us) server_negotiation.cc:299] Negotiation successful
0929 20:32:16.700147 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":275,"threads_started":1}
W20260929 20:32:16.706012 10866 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 36378a5dbd1441d5970d36bf577de0bb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 20:32:16.706178 10866 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 20:32:16.707373 29650 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:32:16.709138 10845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77687716a2c945fa9e927cfcd7e50641: Bootstrap starting.
I20260929 20:32:16.714313 10845 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77687716a2c945fa9e927cfcd7e50641: Neither blocks nor log segments found. Creating new log.
I20260929 20:32:16.716240 10845 log.cc:824] T 00000000000000000000000000000000 P 77687716a2c945fa9e927cfcd7e50641: Log is configured to *not* fsync() on all Append() calls
W20260929 20:32:16.717386 10648 tablet.cc:2369] T 00000000000000000000000000000000 P 36378a5dbd1441d5970d36bf577de0bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:32:16.704916 10627 master_runner.cc:186] Error getting master registration for 127.28.244.189:45623: OK, instance_id { permanent_uuid: "14d03573df38454ba0d41fdf35c31205" instance_seqno: 1790713936027801 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 20:32:16.717810 10627 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 20:32:16.722770 10845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77687716a2c945fa9e927cfcd7e50641: No bootstrap required, opened a new log
I20260929 20:32:16.739010 10845 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77687716a2c945fa9e927cfcd7e50641 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36378a5dbd1441d5970d36bf577de0bb" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34625 } } peers { permanent_uuid: "14d03573df38454ba0d41fdf35c31205" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 45623 } } peers { permanent_uuid: "77687716a2c945fa9e927cfcd7e50641" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 45429 } }
I20260929 20:32:16.739655 10845 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77687716a2c945fa9e927cfcd7e50641 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:32:16.748451 10845 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77687716a2c945fa9e927cfcd7e50641 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77687716a2c945fa9e927cfcd7e50641, State: Initialized, Role: FOLLOWER
I20260929 20:32:16.749825 10845 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77687716a2c945fa9e927cfcd7e50641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36378a5dbd1441d5970d36bf577de0bb" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34625 } } peers { permanent_uuid: "14d03573df38454ba0d41fdf35c31205" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 45623 } } peers { permanent_uuid: "77687716a2c945fa9e927cfcd7e50641" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 45429 } }
I20260929 20:32:16.751188 10849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:16.726460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43012 (local address 127.28.244.190:34625)
0929 20:32:16.726764 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:16.726771 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:16.729276 (+ 2505us) server_negotiation.cc:407] Connection header received
0929 20:32:16.729607 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:16.729614 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:16.729730 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:16.729818 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:16.732582 (+ 2764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.733807 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.736804 (+ 2997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.737550 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.737686 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:16.742634 (+ 4948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:16.742646 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:16.742652 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:16.742759 (+ 107us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:16.747068 (+ 4309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:16.748272 (+ 1204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:16.748280 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:16.748287 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:16.748484 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:16.749015 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:16.749022 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:16.749026 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:16.749477 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:16.749784 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:16.750477 (+ 693us) server_negotiation.cc:299] Negotiation successful
0929 20:32:16.750779 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 20:32:16.760452 10876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77687716a2c945fa9e927cfcd7e50641 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36378a5dbd1441d5970d36bf577de0bb" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34625 } } peers { permanent_uuid: "14d03573df38454ba0d41fdf35c31205" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 45623 } } peers { permanent_uuid: "77687716a2c945fa9e927cfcd7e50641" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 45429 } } }
I20260929 20:32:16.761087 10876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77687716a2c945fa9e927cfcd7e50641 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:32:16.762203 10845 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77687716a2c945fa9e927cfcd7e50641 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:32:16.773154 10854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:16.728326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40140 (local address 127.28.244.189:45623)
0929 20:32:16.728569 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:16.728576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:16.740608 (+ 12032us) server_negotiation.cc:407] Connection header received
0929 20:32:16.740973 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:16.740980 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:16.741072 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:16.741152 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:16.743896 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.745421 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.750736 (+ 5315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.751560 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.751662 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:16.763013 (+ 11351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:16.763021 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:16.763026 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:16.763078 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:16.767313 (+ 4235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:16.768352 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:16.769048 (+ 696us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:16.769055 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:16.769298 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:16.769943 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:16.769951 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:16.769957 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:16.770522 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:16.772065 (+ 1543us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:16.772287 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 20:32:16.772639 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:32:16.775215 10851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:16.740927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35554 (local address 127.28.244.188:45429)
0929 20:32:16.741120 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:16.741128 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:16.752685 (+ 11557us) server_negotiation.cc:407] Connection header received
0929 20:32:16.753004 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:16.753010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:16.753106 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:16.753183 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:16.755469 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.756700 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.760112 (+ 3412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.760933 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.761044 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:16.766076 (+ 5032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:16.766086 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:16.766094 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:16.766179 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:16.770346 (+ 4167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:16.771318 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:16.771324 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:16.771328 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:16.771504 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:16.771987 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:16.771992 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:16.771995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:16.772424 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:16.772747 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:16.774295 (+ 1548us) server_negotiation.cc:299] Negotiation successful
0929 20:32:16.774641 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 20:32:16.776424 10769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.188" port: 45429 } attempt: 5
Sep 29 20:32:16 dist-test-slave-p29l krb5kdc[10614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713935, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.189@KRBTEST.COM for kudu/127.28.244.188@KRBTEST.COM
W20260929 20:32:16.794322 10887 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 77687716a2c945fa9e927cfcd7e50641: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 20:32:16.794538 10887 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 20:32:16.802757 10773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:16.777216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:45429 (local address 127.0.0.1:35566)
0929 20:32:16.777625 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:16.777647 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:16.777778 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:16.778758 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:16.778767 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:16.779248 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:32:16.779516 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:16.779530 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.781190 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.781196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:16.782863 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:16.782889 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.787498 (+ 4609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.787508 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:16.787915 (+ 407us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:16.788796 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:32:16.788829 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:32:16.791904 (+ 3075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:32:16.798684 (+ 6780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:16.798691 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:16.798696 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:16.799295 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:16.799828 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:16.799835 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:16.799839 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:16.800017 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:16.800812 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:32:16.800818 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:32:16.801075 (+ 257us) client_negotiation.cc:769] Sending connection context
0929 20:32:16.801246 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 20:32:16.801596 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258}
I20260929 20:32:16.803964 10851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:16.778028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35566 (local address 127.28.244.188:45429)
0929 20:32:16.778226 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:16.778233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:16.778255 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:32:16.778326 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:16.778332 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:16.778417 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:16.778534 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:16.779705 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.780990 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.785822 (+ 4832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:16.786991 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:16.787191 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:16.794015 (+ 6824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:16.794025 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:16.794031 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:16.794085 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:16.798442 (+ 4357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:16.799463 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:16.799469 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:16.799474 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:16.799659 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:16.800182 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:16.800186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:16.800190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:16.800573 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:16.800728 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:16.803237 (+ 2509us) server_negotiation.cc:299] Negotiation successful
0929 20:32:16.803526 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 20:32:16.787457 10777 master_runner.cc:186] Error getting master registration for 127.28.244.189:45623: OK, instance_id { permanent_uuid: "14d03573df38454ba0d41fdf35c31205" instance_seqno: 1790713936027801 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 20:32:16.806993 10777 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 20:32:16.809544 10769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14d03573df38454ba0d41fdf35c31205: Bootstrap starting.
I20260929 20:32:16.813575 10769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14d03573df38454ba0d41fdf35c31205: Neither blocks nor log segments found. Creating new log.
I20260929 20:32:16.815066 10769 log.cc:824] T 00000000000000000000000000000000 P 14d03573df38454ba0d41fdf35c31205: Log is configured to *not* fsync() on all Append() calls
I20260929 20:32:16.819119 10769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14d03573df38454ba0d41fdf35c31205: No bootstrap required, opened a new log
I20260929 20:32:16.831787 10769 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14d03573df38454ba0d41fdf35c31205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36378a5dbd1441d5970d36bf577de0bb" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34625 } } peers { permanent_uuid: "14d03573df38454ba0d41fdf35c31205" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 45623 } } peers { permanent_uuid: "77687716a2c945fa9e927cfcd7e50641" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 45429 } }
I20260929 20:32:16.832347 10769 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14d03573df38454ba0d41fdf35c31205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:32:16.832486 10769 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14d03573df38454ba0d41fdf35c31205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d03573df38454ba0d41fdf35c31205, State: Initialized, Role: FOLLOWER
I20260929 20:32:16.833455 10769 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14d03573df38454ba0d41fdf35c31205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36378a5dbd1441d5970d36bf577de0bb" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34625 } } peers { permanent_uuid: "14d03573df38454ba0d41fdf35c31205" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 45623 } } peers { permanent_uuid: "77687716a2c945fa9e927cfcd7e50641" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 45429 } }
I20260929 20:32:16.834743 10890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14d03573df38454ba0d41fdf35c31205 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36378a5dbd1441d5970d36bf577de0bb" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34625 } } peers { permanent_uuid: "14d03573df38454ba0d41fdf35c31205" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 45623 } } peers { permanent_uuid: "77687716a2c945fa9e927cfcd7e50641" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 45429 } } }
I20260929 20:32:16.835309 10890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14d03573df38454ba0d41fdf35c31205 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:32:16.836304 10769 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14d03573df38454ba0d41fdf35c31205 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 20:32:16.844506 10799 tablet.cc:2369] T 00000000000000000000000000000000 P 77687716a2c945fa9e927cfcd7e50641: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:32:16.864917 10702 master_runner.cc:263] No leader master found from master 14d03573df38454ba0d41fdf35c31205
I20260929 20:32:16.865104 10702 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 20:32:16.866489 10901 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 14d03573df38454ba0d41fdf35c31205: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 20:32:16.866641 10901 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 20:32:16.960758 10890 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14d03573df38454ba0d41fdf35c31205 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:32:16.961095 10890 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14d03573df38454ba0d41fdf35c31205 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36378a5dbd1441d5970d36bf577de0bb" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34625 } } peers { permanent_uuid: "14d03573df38454ba0d41fdf35c31205" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 45623 } } peers { permanent_uuid: "77687716a2c945fa9e927cfcd7e50641" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 45429 } }
I20260929 20:32:16.962983 10890 leader_election.cc:288] T 00000000000000000000000000000000 P 14d03573df38454ba0d41fdf35c31205 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36378a5dbd1441d5970d36bf577de0bb (127.28.244.190:34625), 77687716a2c945fa9e927cfcd7e50641 (127.28.244.188:45429)
I20260929 20:32:16.964215 10670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "14d03573df38454ba0d41fdf35c31205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36378a5dbd1441d5970d36bf577de0bb" is_pre_election: true
I20260929 20:32:16.964915 10670 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 36378a5dbd1441d5970d36bf577de0bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14d03573df38454ba0d41fdf35c31205 in term 0.
I20260929 20:32:16.966208 10820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "14d03573df38454ba0d41fdf35c31205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77687716a2c945fa9e927cfcd7e50641" is_pre_election: true
I20260929 20:32:16.966871 10820 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 77687716a2c945fa9e927cfcd7e50641 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14d03573df38454ba0d41fdf35c31205 in term 0.
I20260929 20:32:16.967584 10719 leader_election.cc:302] T 00000000000000000000000000000000 P 14d03573df38454ba0d41fdf35c31205 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14d03573df38454ba0d41fdf35c31205, 77687716a2c945fa9e927cfcd7e50641; no voters:
I20260929 20:32:16.968499 10890 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14d03573df38454ba0d41fdf35c31205 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:32:16.968667 10890 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14d03573df38454ba0d41fdf35c31205 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:32:16.968791 10890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14d03573df38454ba0d41fdf35c31205 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:32:16.973297 10890 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14d03573df38454ba0d41fdf35c31205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36378a5dbd1441d5970d36bf577de0bb" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34625 } } peers { permanent_uuid: "14d03573df38454ba0d41fdf35c31205" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 45623 } } peers { permanent_uuid: "77687716a2c945fa9e927cfcd7e50641" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 45429 } }
I20260929 20:32:16.976743 10890 leader_election.cc:288] T 00000000000000000000000000000000 P 14d03573df38454ba0d41fdf35c31205 [CANDIDATE]: Term 1 election: Requested vote from peers 36378a5dbd1441d5970d36bf577de0bb (127.28.244.190:34625), 77687716a2c945fa9e927cfcd7e50641 (127.28.244.188:45429)
I20260929 20:32:16.977600 10670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "14d03573df38454ba0d41fdf35c31205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36378a5dbd1441d5970d36bf577de0bb"
I20260929 20:32:16.977910 10670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36378a5dbd1441d5970d36bf577de0bb [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:32:16.981968 10670 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 36378a5dbd1441d5970d36bf577de0bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14d03573df38454ba0d41fdf35c31205 in term 1.
I20260929 20:32:16.983047 10820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "14d03573df38454ba0d41fdf35c31205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77687716a2c945fa9e927cfcd7e50641"
I20260929 20:32:16.983343 10820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77687716a2c945fa9e927cfcd7e50641 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:32:16.987118 10820 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 77687716a2c945fa9e927cfcd7e50641 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14d03573df38454ba0d41fdf35c31205 in term 1.
I20260929 20:32:16.988343 10719 leader_election.cc:302] T 00000000000000000000000000000000 P 14d03573df38454ba0d41fdf35c31205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14d03573df38454ba0d41fdf35c31205, 77687716a2c945fa9e927cfcd7e50641; no voters:
I20260929 20:32:16.988952 10890 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14d03573df38454ba0d41fdf35c31205 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:32:17.000933 10890 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14d03573df38454ba0d41fdf35c31205 [term 1 LEADER]: Becoming Leader. State: Replica: 14d03573df38454ba0d41fdf35c31205, State: Running, Role: LEADER
I20260929 20:32:17.001688 10890 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14d03573df38454ba0d41fdf35c31205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36378a5dbd1441d5970d36bf577de0bb" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34625 } } peers { permanent_uuid: "14d03573df38454ba0d41fdf35c31205" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 45623 } } peers { permanent_uuid: "77687716a2c945fa9e927cfcd7e50641" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 45429 } }
I20260929 20:32:17.011454 10903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14d03573df38454ba0d41fdf35c31205 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14d03573df38454ba0d41fdf35c31205. Latest consensus state: current_term: 1 leader_uuid: "14d03573df38454ba0d41fdf35c31205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36378a5dbd1441d5970d36bf577de0bb" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34625 } } peers { permanent_uuid: "14d03573df38454ba0d41fdf35c31205" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 45623 } } peers { permanent_uuid: "77687716a2c945fa9e927cfcd7e50641" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 45429 } } }
I20260929 20:32:17.012001 10903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14d03573df38454ba0d41fdf35c31205 [sys.catalog]: This master's current role is: LEADER
W20260929 20:32:17.017482 10723 tablet.cc:2369] T 00000000000000000000000000000000 P 14d03573df38454ba0d41fdf35c31205: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:32:17.018304 10905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:32:17.020944 10905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:32:17.072395 10820 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 77687716a2c945fa9e927cfcd7e50641 [term 1 FOLLOWER]: Refusing update from remote peer 14d03573df38454ba0d41fdf35c31205: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:32:17.073807 10903 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 14d03573df38454ba0d41fdf35c31205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77687716a2c945fa9e927cfcd7e50641" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 45429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:32:17.082670 10670 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 36378a5dbd1441d5970d36bf577de0bb [term 1 FOLLOWER]: Refusing update from remote peer 14d03573df38454ba0d41fdf35c31205: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:32:17.083797 10903 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 14d03573df38454ba0d41fdf35c31205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36378a5dbd1441d5970d36bf577de0bb" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:32:17.122644 10855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36378a5dbd1441d5970d36bf577de0bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14d03573df38454ba0d41fdf35c31205. Latest consensus state: current_term: 1 leader_uuid: "14d03573df38454ba0d41fdf35c31205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36378a5dbd1441d5970d36bf577de0bb" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34625 } } peers { permanent_uuid: "14d03573df38454ba0d41fdf35c31205" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 45623 } } peers { permanent_uuid: "77687716a2c945fa9e927cfcd7e50641" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 45429 } } }
I20260929 20:32:17.123023 10855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36378a5dbd1441d5970d36bf577de0bb [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:32:17.127754 10906 mvcc.cc:205] Tried to move back new op lower bound from 7334764286232748032 to 7334764285964853248. Current Snapshot: MvccSnapshot[applied={T|T < 7334764286232748032}]
I20260929 20:32:17.131846 10876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77687716a2c945fa9e927cfcd7e50641 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14d03573df38454ba0d41fdf35c31205. Latest consensus state: current_term: 1 leader_uuid: "14d03573df38454ba0d41fdf35c31205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36378a5dbd1441d5970d36bf577de0bb" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34625 } } peers { permanent_uuid: "14d03573df38454ba0d41fdf35c31205" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 45623 } } peers { permanent_uuid: "77687716a2c945fa9e927cfcd7e50641" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 45429 } } }
I20260929 20:32:17.132225 10876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77687716a2c945fa9e927cfcd7e50641 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:32:17.134372 10908 mvcc.cc:205] Tried to move back new op lower bound from 7334764286232748032 to 7334764285964853248. Current Snapshot: MvccSnapshot[applied={T|T < 7334764286232748032}]
I20260929 20:32:17.139900 10907 mvcc.cc:205] Tried to move back new op lower bound from 7334764286232748032 to 7334764285964853248. Current Snapshot: MvccSnapshot[applied={T|T < 7334764286232748032}]
I20260929 20:32:17.145795 10855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36378a5dbd1441d5970d36bf577de0bb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "14d03573df38454ba0d41fdf35c31205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36378a5dbd1441d5970d36bf577de0bb" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34625 } } peers { permanent_uuid: "14d03573df38454ba0d41fdf35c31205" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 45623 } } peers { permanent_uuid: "77687716a2c945fa9e927cfcd7e50641" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 45429 } } }
I20260929 20:32:17.146219 10855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36378a5dbd1441d5970d36bf577de0bb [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:32:17.150838 10876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77687716a2c945fa9e927cfcd7e50641 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "14d03573df38454ba0d41fdf35c31205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36378a5dbd1441d5970d36bf577de0bb" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34625 } } peers { permanent_uuid: "14d03573df38454ba0d41fdf35c31205" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 45623 } } peers { permanent_uuid: "77687716a2c945fa9e927cfcd7e50641" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 45429 } } }
I20260929 20:32:17.151158 10876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77687716a2c945fa9e927cfcd7e50641 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:32:17.151880 10903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14d03573df38454ba0d41fdf35c31205 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "14d03573df38454ba0d41fdf35c31205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36378a5dbd1441d5970d36bf577de0bb" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34625 } } peers { permanent_uuid: "14d03573df38454ba0d41fdf35c31205" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 45623 } } peers { permanent_uuid: "77687716a2c945fa9e927cfcd7e50641" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 45429 } } }
I20260929 20:32:17.152261 10903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14d03573df38454ba0d41fdf35c31205 [sys.catalog]: This master's current role is: LEADER
I20260929 20:32:17.152621 10903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14d03573df38454ba0d41fdf35c31205 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "14d03573df38454ba0d41fdf35c31205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36378a5dbd1441d5970d36bf577de0bb" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34625 } } peers { permanent_uuid: "14d03573df38454ba0d41fdf35c31205" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 45623 } } peers { permanent_uuid: "77687716a2c945fa9e927cfcd7e50641" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 45429 } } }
I20260929 20:32:17.152863 10903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14d03573df38454ba0d41fdf35c31205 [sys.catalog]: This master's current role is: LEADER
I20260929 20:32:17.154692 10905 catalog_manager.cc:1347] Generated new cluster ID: 164f0fa6fe494032a9068a6584443d97
I20260929 20:32:17.154800 10905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:32:17.199303 10905 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:32:17.212452 10905 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:32:17.241748 10905 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14d03573df38454ba0d41fdf35c31205: Generated new TSK 0
I20260929 20:32:17.243237 10905 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:32:17.708489 10866 catalog_manager.cc:1259] Loaded cluster ID: 164f0fa6fe494032a9068a6584443d97
I20260929 20:32:17.708603 10866 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 36378a5dbd1441d5970d36bf577de0bb: loading cluster ID for follower catalog manager: success
I20260929 20:32:17.711700 10866 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 36378a5dbd1441d5970d36bf577de0bb: acquiring CA information for follower catalog manager: success
I20260929 20:32:17.713846 10866 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 36378a5dbd1441d5970d36bf577de0bb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 20:32:17.796922 10887 catalog_manager.cc:1259] Loaded cluster ID: 164f0fa6fe494032a9068a6584443d97
I20260929 20:32:17.797075 10887 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 77687716a2c945fa9e927cfcd7e50641: loading cluster ID for follower catalog manager: success
I20260929 20:32:17.799958 10887 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 77687716a2c945fa9e927cfcd7e50641: acquiring CA information for follower catalog manager: success
I20260929 20:32:17.802532 10887 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 77687716a2c945fa9e927cfcd7e50641: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 20:32:17.880191 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 10627
I20260929 20:32:17.881366 10690 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:32:17.996481 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 10627
I20260929 20:32:18.018795 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 10702
I20260929 20:32:18.020196 10764 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:32:18.130959 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 10702
I20260929 20:32:18.155793 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 10777
I20260929 20:32:18.157169 10840 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:32:18.271589 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 10777
2026-09-29T20:32:18Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4710 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:32:18 dist-test-slave-p29l krb5kdc[10923](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:32:18 dist-test-slave-p29l krb5kdc[10923](info): set up 2 sockets
Sep 29 20:32:18 dist-test-slave-p29l krb5kdc[10923](info): commencing operation
krb5kdc: starting...
W20260929 20:32:19.403965 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:32:19 dist-test-slave-p29l krb5kdc[10923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:32:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:32:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.244.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
I20260929 20:32:19.681038 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:41861
--webserver_interface=127.28.244.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:44539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:41861
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:32:20.063097 10937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:32:20.063434 10937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:32:20.063611 10937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:32:20.071528 10937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:32:20.071684 10937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:32:20.071775 10937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:32:20.071897 10937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:32:20.071942 10937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:32:20.083096 10937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:44539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:41861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:41861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:32:20.084828 10937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:32:20.086454 10937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:32:20.100633 10942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:20.100523 10943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:20.100809 10945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:32:20.101357 10937 server_base.cc:1034] running on GCE node
I20260929 20:32:20.102411 10937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:32:20.104246 10937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:32:20.105567 10937 hybrid_clock.cc:648] HybridClock initialized: now 1790713940105487 us; error 61 us; skew 500 ppm
I20260929 20:32:20.106065 10937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:32:20 dist-test-slave-p29l krb5kdc[10923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713940, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:32:20.110186 10937 init.cc:377] Logged in from keytab as kudu/127.28.244.190@KRBTEST.COM (short username kudu)
I20260929 20:32:20.112377 10937 webserver.cc:460] Webserver started at http://127.28.244.190:34653/ using document root <none> and password file <none>
I20260929 20:32:20.113287 10937 fs_manager.cc:359] Metadata directory not provided
I20260929 20:32:20.113461 10937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:32:20.113808 10937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:32:20.117390 10937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "cf84ca31197b40c0a644657bc3d2d16d"
format_stamp: "Formatted at 2026-09-29 20:32:20 on dist-test-slave-p29l"
server_key: "417bf8cdf535d557689aec624d44f980"
server_key_iv: "ab6e408876a1a6196afc9724c607c1d8"
server_key_version: "encryptionkey@0"
I20260929 20:32:20.118345 10937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "cf84ca31197b40c0a644657bc3d2d16d"
format_stamp: "Formatted at 2026-09-29 20:32:20 on dist-test-slave-p29l"
server_key: "417bf8cdf535d557689aec624d44f980"
server_key_iv: "ab6e408876a1a6196afc9724c607c1d8"
server_key_version: "encryptionkey@0"
I20260929 20:32:20.125043 10937 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260929 20:32:20.129554 10952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:20.131795 10937 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 20:32:20.132120 10937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "cf84ca31197b40c0a644657bc3d2d16d"
format_stamp: "Formatted at 2026-09-29 20:32:20 on dist-test-slave-p29l"
server_key: "417bf8cdf535d557689aec624d44f980"
server_key_iv: "ab6e408876a1a6196afc9724c607c1d8"
server_key_version: "encryptionkey@0"
I20260929 20:32:20.132524 10937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:32:20.155157 10937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:32:20.162947 10937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:32:20.163513 10937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:32:20.186729 10937 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:41861
I20260929 20:32:20.186764 11003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:41861 every 8 connection(s)
I20260929 20:32:20.188398 10937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:32:20.193612 11004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:32:20.197737 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 10937
I20260929 20:32:20.198071 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:32:20.198813 29650 external_mini_cluster.cc:1428] Setting key 6b51d2e7df1fff7d42b0c648676ed3aa
I20260929 20:32:20.207605 11004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf84ca31197b40c0a644657bc3d2d16d: Bootstrap starting.
I20260929 20:32:20.211277 11004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf84ca31197b40c0a644657bc3d2d16d: Neither blocks nor log segments found. Creating new log.
I20260929 20:32:20.212566 11004 log.cc:824] T 00000000000000000000000000000000 P cf84ca31197b40c0a644657bc3d2d16d: Log is configured to *not* fsync() on all Append() calls
Sep 29 20:32:20 dist-test-slave-p29l krb5kdc[10923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:32:20.217065 11004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf84ca31197b40c0a644657bc3d2d16d: No bootstrap required, opened a new log
I20260929 20:32:20.226116 11004 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf84ca31197b40c0a644657bc3d2d16d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf84ca31197b40c0a644657bc3d2d16d" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 41861 } }
I20260929 20:32:20.226583 11004 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf84ca31197b40c0a644657bc3d2d16d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:32:20.226742 11004 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf84ca31197b40c0a644657bc3d2d16d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf84ca31197b40c0a644657bc3d2d16d, State: Initialized, Role: FOLLOWER
I20260929 20:32:20.227428 11004 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf84ca31197b40c0a644657bc3d2d16d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf84ca31197b40c0a644657bc3d2d16d" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 41861 } }
I20260929 20:32:20.227723 11004 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf84ca31197b40c0a644657bc3d2d16d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:32:20.227937 11004 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf84ca31197b40c0a644657bc3d2d16d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:32:20.228140 11004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf84ca31197b40c0a644657bc3d2d16d [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:32:20.228227 11007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:20.200820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44238 (local address 127.28.244.190:41861)
0929 20:32:20.203095 (+ 2275us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:20.203117 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:20.203184 (+ 67us) server_negotiation.cc:407] Connection header received
0929 20:32:20.206218 (+ 3034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:20.206252 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:20.206791 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:20.207315 (+ 524us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:20.209049 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:20.210479 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:20.212170 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:20.213170 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:20.213465 (+ 295us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:20.218516 (+ 5051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:20.218526 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:20.218544 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:20.218593 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:20.222923 (+ 4330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:20.223995 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:20.224004 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:20.224012 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:20.224233 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:20.224712 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:20.224720 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:20.224722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:20.225394 (+ 672us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:20.225745 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:20.226223 (+ 478us) server_negotiation.cc:299] Negotiation successful
0929 20:32:20.226777 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1957,"thread_start_us":1657,"threads_started":1}
I20260929 20:32:20.231593 11004 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf84ca31197b40c0a644657bc3d2d16d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf84ca31197b40c0a644657bc3d2d16d" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 41861 } }
I20260929 20:32:20.232275 11004 leader_election.cc:302] T 00000000000000000000000000000000 P cf84ca31197b40c0a644657bc3d2d16d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf84ca31197b40c0a644657bc3d2d16d; no voters:
I20260929 20:32:20.233088 11004 leader_election.cc:288] T 00000000000000000000000000000000 P cf84ca31197b40c0a644657bc3d2d16d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:32:20.233276 11009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf84ca31197b40c0a644657bc3d2d16d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:32:20.233506 29650 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:32:20.235443 11004 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf84ca31197b40c0a644657bc3d2d16d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:32:20.236222 11009 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf84ca31197b40c0a644657bc3d2d16d [term 1 LEADER]: Becoming Leader. State: Replica: cf84ca31197b40c0a644657bc3d2d16d, State: Running, Role: LEADER
I20260929 20:32:20.237119 11009 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf84ca31197b40c0a644657bc3d2d16d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf84ca31197b40c0a644657bc3d2d16d" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 41861 } }
I20260929 20:32:20.241022 11011 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf84ca31197b40c0a644657bc3d2d16d [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf84ca31197b40c0a644657bc3d2d16d. Latest consensus state: current_term: 1 leader_uuid: "cf84ca31197b40c0a644657bc3d2d16d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf84ca31197b40c0a644657bc3d2d16d" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 41861 } } }
I20260929 20:32:20.241572 11011 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf84ca31197b40c0a644657bc3d2d16d [sys.catalog]: This master's current role is: LEADER
I20260929 20:32:20.241809 11010 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf84ca31197b40c0a644657bc3d2d16d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf84ca31197b40c0a644657bc3d2d16d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf84ca31197b40c0a644657bc3d2d16d" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 41861 } } }
I20260929 20:32:20.242274 11010 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf84ca31197b40c0a644657bc3d2d16d [sys.catalog]: This master's current role is: LEADER
I20260929 20:32:20.243530 11017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:32:20.248703 11017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:32:20.258765 11017 catalog_manager.cc:1347] Generated new cluster ID: 377332020bf240898a7a6d4b570b8589
I20260929 20:32:20.258921 11017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:32:20.281136 11017 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:32:20.282704 11017 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:32:20.293773 11017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf84ca31197b40c0a644657bc3d2d16d: Generated new TSK 0
I20260929 20:32:20.294713 11017 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:32:20.350876 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 10937
I20260929 20:32:20.352487 10999 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:32:20.456810 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 10937
2026-09-29T20:32:20Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2179 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:32:20 dist-test-slave-p29l krb5kdc[11032](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:32:20 dist-test-slave-p29l krb5kdc[11032](info): set up 2 sockets
Sep 29 20:32:20 dist-test-slave-p29l krb5kdc[11032](info): commencing operation
krb5kdc: starting...
W20260929 20:32:21.600075 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:32:21 dist-test-slave-p29l krb5kdc[11032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:32:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:32:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.244.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
I20260929 20:32:21.864583 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:37845
--webserver_interface=127.28.244.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:34391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:37845
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:32:22.237800 11045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:32:22.238152 11045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:32:22.238324 11045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:32:22.246289 11045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:32:22.246438 11045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:32:22.246518 11045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:32:22.246626 11045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:32:22.246690 11045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:32:22.257714 11045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:34391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:37845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:37845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:32:22.259397 11045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:32:22.260908 11045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:32:22.275864 11053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:22.275887 11050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:22.275784 11051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:32:22.276968 11045 server_base.cc:1034] running on GCE node
I20260929 20:32:22.278038 11045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:32:22.279923 11045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:32:22.281237 11045 hybrid_clock.cc:648] HybridClock initialized: now 1790713942281174 us; error 53 us; skew 500 ppm
I20260929 20:32:22.281759 11045 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:32:22 dist-test-slave-p29l krb5kdc[11032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713942, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:32:22.285864 11045 init.cc:377] Logged in from keytab as kudu/127.28.244.190@KRBTEST.COM (short username kudu)
I20260929 20:32:22.287874 11045 webserver.cc:460] Webserver started at http://127.28.244.190:40263/ using document root <none> and password file <none>
I20260929 20:32:22.288722 11045 fs_manager.cc:359] Metadata directory not provided
I20260929 20:32:22.288899 11045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:32:22.289249 11045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:32:22.293149 11045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "7419d303241e4b149f92d21d7bc9132a"
format_stamp: "Formatted at 2026-09-29 20:32:22 on dist-test-slave-p29l"
server_key: "4b0c1ac5b33c319c2789b0ab2aad0838"
server_key_iv: "9561148bfb031b03b0d3b191f5ee30e1"
server_key_version: "encryptionkey@0"
I20260929 20:32:22.294138 11045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "7419d303241e4b149f92d21d7bc9132a"
format_stamp: "Formatted at 2026-09-29 20:32:22 on dist-test-slave-p29l"
server_key: "4b0c1ac5b33c319c2789b0ab2aad0838"
server_key_iv: "9561148bfb031b03b0d3b191f5ee30e1"
server_key_version: "encryptionkey@0"
I20260929 20:32:22.300989 11045 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 20:32:22.310791 11060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:22.313126 11045 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.004s
I20260929 20:32:22.313395 11045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "7419d303241e4b149f92d21d7bc9132a"
format_stamp: "Formatted at 2026-09-29 20:32:22 on dist-test-slave-p29l"
server_key: "4b0c1ac5b33c319c2789b0ab2aad0838"
server_key_iv: "9561148bfb031b03b0d3b191f5ee30e1"
server_key_version: "encryptionkey@0"
I20260929 20:32:22.313685 11045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:32:22.343178 11045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:32:22.348333 11045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:32:22.348661 11045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:32:22.369417 11045 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:37845
I20260929 20:32:22.369447 11112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:37845 every 8 connection(s)
I20260929 20:32:22.371121 11045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:32:22.371610 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 11045
I20260929 20:32:22.371879 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:32:22.372853 29650 external_mini_cluster.cc:1428] Setting key 612630ef99161bb60da39a8100872212
I20260929 20:32:22.376295 11113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 20:32:22 dist-test-slave-p29l krb5kdc[11032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:32:22.393064 11113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a: Bootstrap starting.
I20260929 20:32:22.397996 11113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a: Neither blocks nor log segments found. Creating new log.
I20260929 20:32:22.399367 11113 log.cc:824] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a: Log is configured to *not* fsync() on all Append() calls
I20260929 20:32:22.401625 11116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:22.375012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45824 (local address 127.28.244.190:37845)
0929 20:32:22.375952 (+ 940us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:22.375966 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:22.376011 (+ 45us) server_negotiation.cc:407] Connection header received
0929 20:32:22.377782 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:22.377808 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:22.378250 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:22.378643 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:22.380882 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:22.382718 (+ 1836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:22.385066 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:22.385899 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:22.386068 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:22.391680 (+ 5612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:22.391695 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:22.391719 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:22.391788 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:22.396565 (+ 4777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:22.397475 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:22.397487 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:22.397498 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:22.397769 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:22.398228 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:22.398239 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:22.398243 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:22.398989 (+ 746us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:22.399391 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:22.399933 (+ 542us) server_negotiation.cc:299] Negotiation successful
0929 20:32:22.400509 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":413,"threads_started":1}
I20260929 20:32:22.403144 11113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a: No bootstrap required, opened a new log
I20260929 20:32:22.412923 11113 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7419d303241e4b149f92d21d7bc9132a" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 37845 } }
I20260929 20:32:22.413384 11113 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:32:22.413468 11113 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7419d303241e4b149f92d21d7bc9132a, State: Initialized, Role: FOLLOWER
I20260929 20:32:22.414317 11113 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7419d303241e4b149f92d21d7bc9132a" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 37845 } }
I20260929 20:32:22.414584 11113 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:32:22.414713 11113 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:32:22.414865 11113 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:32:22.417508 11113 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7419d303241e4b149f92d21d7bc9132a" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 37845 } }
I20260929 20:32:22.418355 11113 leader_election.cc:302] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7419d303241e4b149f92d21d7bc9132a; no voters:
I20260929 20:32:22.419355 11113 leader_election.cc:288] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:32:22.419540 11118 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:32:22.420356 11118 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [term 1 LEADER]: Becoming Leader. State: Replica: 7419d303241e4b149f92d21d7bc9132a, State: Running, Role: LEADER
I20260929 20:32:22.421001 11118 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7419d303241e4b149f92d21d7bc9132a" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 37845 } }
I20260929 20:32:22.421926 11113 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:32:22.424347 11119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7419d303241e4b149f92d21d7bc9132a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7419d303241e4b149f92d21d7bc9132a" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 37845 } } }
I20260929 20:32:22.424324 11120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7419d303241e4b149f92d21d7bc9132a. Latest consensus state: current_term: 1 leader_uuid: "7419d303241e4b149f92d21d7bc9132a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7419d303241e4b149f92d21d7bc9132a" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 37845 } } }
I20260929 20:32:22.424832 11119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [sys.catalog]: This master's current role is: LEADER
I20260929 20:32:22.424871 11120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [sys.catalog]: This master's current role is: LEADER
I20260929 20:32:22.426683 11124 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:32:22.433156 11124 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:32:22.443991 11124 catalog_manager.cc:1347] Generated new cluster ID: bc694c4e95ae41cebabaea68f9359e61
I20260929 20:32:22.444139 11124 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:32:22.460791 11124 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:32:22.462543 11124 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:32:22.474977 11124 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a: Generated new TSK 0
I20260929 20:32:22.476119 11124 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.244.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab.
Entry for principal kudu/127.28.244.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab.
I20260929 20:32:22.552716 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.129:0
--local_ip_for_outbound_sockets=127.28.244.129
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:37845
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:34391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:32:22.939034 11139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:32:22.939359 11139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:32:22.939515 11139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:32:22.947467 11139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:32:22.947614 11139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:32:22.947841 11139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.129
I20260929 20:32:22.958588 11139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:34391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:37845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.129
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:32:22.960177 11139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:32:22.961643 11139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:32:22.977386 11144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:22.977365 11145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:22.977667 11147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:32:22.984220 11139 server_base.cc:1034] running on GCE node
I20260929 20:32:22.985025 11139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:32:22.986191 11139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:32:22.987473 11139 hybrid_clock.cc:648] HybridClock initialized: now 1790713942987414 us; error 55 us; skew 500 ppm
I20260929 20:32:22.987941 11139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:32:22 dist-test-slave-p29l krb5kdc[11032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713942, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:32:22.992005 11139 init.cc:377] Logged in from keytab as kudu/127.28.244.129@KRBTEST.COM (short username kudu)
I20260929 20:32:22.994307 11139 webserver.cc:460] Webserver started at http://127.28.244.129:45621/ using document root <none> and password file <none>
I20260929 20:32:22.995234 11139 fs_manager.cc:359] Metadata directory not provided
I20260929 20:32:22.995411 11139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:32:22.995775 11139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:32:22.999668 11139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/data/instance:
uuid: "e1aa120e44bb45e68986e21a8565480e"
format_stamp: "Formatted at 2026-09-29 20:32:22 on dist-test-slave-p29l"
server_key: "897c92fc99c51a8554998a2cb5ad9653"
server_key_iv: "6795ee87173cd6dd9f3872ce37b369af"
server_key_version: "encryptionkey@0"
I20260929 20:32:23.000927 11139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance:
uuid: "e1aa120e44bb45e68986e21a8565480e"
format_stamp: "Formatted at 2026-09-29 20:32:22 on dist-test-slave-p29l"
server_key: "897c92fc99c51a8554998a2cb5ad9653"
server_key_iv: "6795ee87173cd6dd9f3872ce37b369af"
server_key_version: "encryptionkey@0"
I20260929 20:32:23.007368 11139 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 20:32:23.011662 11154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:23.013482 11139 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 20:32:23.013727 11139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/wal
uuid: "e1aa120e44bb45e68986e21a8565480e"
format_stamp: "Formatted at 2026-09-29 20:32:22 on dist-test-slave-p29l"
server_key: "897c92fc99c51a8554998a2cb5ad9653"
server_key_iv: "6795ee87173cd6dd9f3872ce37b369af"
server_key_version: "encryptionkey@0"
I20260929 20:32:23.013971 11139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:32:23.038266 11139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:32:23.043869 11139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:32:23.044234 11139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:32:23.045542 11139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:32:23.047241 11139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:32:23.047360 11139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:23.047518 11139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:32:23.047590 11139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:23.089380 11139 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.129:33535
I20260929 20:32:23.089437 11266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.129:33535 every 8 connection(s)
I20260929 20:32:23.090821 11139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
I20260929 20:32:23.092041 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 11139
I20260929 20:32:23.092259 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance
I20260929 20:32:23.093127 29650 external_mini_cluster.cc:1428] Setting key a356b8d6b3ef30af7eb3a0069f87bc79
Sep 29 20:32:23 dist-test-slave-p29l krb5kdc[11032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713942, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:32:23.123090 11269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:23.094510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:56479 (local address 127.28.244.190:37845)
0929 20:32:23.095028 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:23.095033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:23.096523 (+ 1490us) server_negotiation.cc:407] Connection header received
0929 20:32:23.100185 (+ 3662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:23.100191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:23.100279 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:23.100392 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:23.104088 (+ 3696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:23.104873 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:23.107477 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:23.108314 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:23.108492 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:23.113542 (+ 5050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:23.113552 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:23.113560 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:23.113612 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:23.117727 (+ 4115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:23.119155 (+ 1428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:23.119165 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:23.119171 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:23.119389 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:23.120204 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:23.120212 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:23.120219 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:23.120702 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:23.120907 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:23.121817 (+ 910us) server_negotiation.cc:299] Negotiation successful
0929 20:32:23.122122 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":300,"threads_started":1}
I20260929 20:32:23.124126 11270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:23.094975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.129:56479)
0929 20:32:23.096289 (+ 1314us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:23.096358 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:23.099636 (+ 3278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:23.100773 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:23.100792 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:23.102710 (+ 1918us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:32:23.103738 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:23.103773 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:23.105202 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:23.105212 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:23.107211 (+ 1999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:23.107232 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:23.108620 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:23.108627 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:23.108924 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:23.109724 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:32:23.110243 (+ 519us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:32:23.113202 (+ 2959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:32:23.118007 (+ 4805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:23.118017 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:23.118036 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:23.118939 (+ 903us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:23.119577 (+ 638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:23.119582 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:23.119585 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:23.119976 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:23.120888 (+ 912us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:32:23.120896 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:32:23.121345 (+ 449us) client_negotiation.cc:769] Sending connection context
0929 20:32:23.121619 (+ 274us) client_negotiation.cc:240] Negotiation successful
0929 20:32:23.122253 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":919,"thread_start_us":541,"threads_started":1}
WARNING: no policy specified for kudu/127.28.244.130@KRBTEST.COM; defaulting to no policy
I20260929 20:32:23.126012 11267 heartbeater.cc:347] Connected to a master server at 127.28.244.190:37845
I20260929 20:32:23.126526 11267 heartbeater.cc:464] Registering TS with master...
I20260929 20:32:23.127456 11267 heartbeater.cc:511] Master 127.28.244.190:37845 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.130@KRBTEST.COM" created.
I20260929 20:32:23.130417 11078 ts_manager.cc:194] Registered new tserver with Master: e1aa120e44bb45e68986e21a8565480e (127.28.244.129:33535)
I20260929 20:32:23.132671 11078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.129@KRBTEST.COM'} at 127.28.244.129:56479
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab.
Entry for principal kudu/127.28.244.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab.
I20260929 20:32:23.161486 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.130:0
--local_ip_for_outbound_sockets=127.28.244.130
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:37845
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:34391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:32:23.572495 11273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:32:23.572849 11273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:32:23.573032 11273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:32:23.582846 11273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:32:23.582996 11273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:32:23.583189 11273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.130
I20260929 20:32:23.593905 11273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:34391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:37845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.130
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:32:23.595405 11273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:32:23.596941 11273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:32:23.613299 11278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:23.613204 11279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:23.613195 11281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:32:23.620453 11273 server_base.cc:1034] running on GCE node
I20260929 20:32:23.621399 11273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:32:23.622817 11273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:32:23.624241 11273 hybrid_clock.cc:648] HybridClock initialized: now 1790713943624181 us; error 47 us; skew 500 ppm
I20260929 20:32:23.624825 11273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:32:23 dist-test-slave-p29l krb5kdc[11032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713943, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:32:23.629922 11273 init.cc:377] Logged in from keytab as kudu/127.28.244.130@KRBTEST.COM (short username kudu)
I20260929 20:32:23.632215 11273 webserver.cc:460] Webserver started at http://127.28.244.130:45643/ using document root <none> and password file <none>
I20260929 20:32:23.633144 11273 fs_manager.cc:359] Metadata directory not provided
I20260929 20:32:23.633318 11273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:32:23.633709 11273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:32:23.637748 11273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/data/instance:
uuid: "8312b6a9e28542de82dc41db65e09a35"
format_stamp: "Formatted at 2026-09-29 20:32:23 on dist-test-slave-p29l"
server_key: "76df5c2f91d50add72b4b3d9dba917bf"
server_key_iv: "426b91133a80b2d5b8787c24f1163ab6"
server_key_version: "encryptionkey@0"
I20260929 20:32:23.638767 11273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance:
uuid: "8312b6a9e28542de82dc41db65e09a35"
format_stamp: "Formatted at 2026-09-29 20:32:23 on dist-test-slave-p29l"
server_key: "76df5c2f91d50add72b4b3d9dba917bf"
server_key_iv: "426b91133a80b2d5b8787c24f1163ab6"
server_key_version: "encryptionkey@0"
I20260929 20:32:23.645285 11273 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260929 20:32:23.650465 11288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:23.652316 11273 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 20:32:23.652567 11273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/wal
uuid: "8312b6a9e28542de82dc41db65e09a35"
format_stamp: "Formatted at 2026-09-29 20:32:23 on dist-test-slave-p29l"
server_key: "76df5c2f91d50add72b4b3d9dba917bf"
server_key_iv: "426b91133a80b2d5b8787c24f1163ab6"
server_key_version: "encryptionkey@0"
I20260929 20:32:23.652823 11273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:32:23.670174 11273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:32:23.675562 11273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:32:23.675967 11273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:32:23.677079 11273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:32:23.678761 11273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:32:23.678889 11273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:23.679051 11273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:32:23.679131 11273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:23.720561 11273 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.130:40679
I20260929 20:32:23.720661 11402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.130:40679 every 8 connection(s)
I20260929 20:32:23.721966 11273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
I20260929 20:32:23.732229 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 11273
I20260929 20:32:23.732445 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance
I20260929 20:32:23.732960 29650 external_mini_cluster.cc:1428] Setting key 5cf57605bbff20f7589e99f3f1833d95
Sep 29 20:32:23 dist-test-slave-p29l krb5kdc[11032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713943, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:32:23.748643 11405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:23.725031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:58277 (local address 127.28.244.190:37845)
0929 20:32:23.725551 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:23.725559 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:23.726829 (+ 1270us) server_negotiation.cc:407] Connection header received
0929 20:32:23.729023 (+ 2194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:23.729031 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:23.729144 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:23.729300 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:23.731364 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:23.732382 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:23.734768 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:23.735437 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:23.735590 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:23.740097 (+ 4507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:23.740109 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:23.740118 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:23.740178 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:23.744379 (+ 4201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:23.745677 (+ 1298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:23.745682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:23.745685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:23.745846 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:23.746498 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:23.746503 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:23.746505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:23.746856 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:23.747004 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:23.748108 (+ 1104us) server_negotiation.cc:299] Negotiation successful
0929 20:32:23.748354 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":256,"threads_started":1}
I20260929 20:32:23.750419 11406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:23.725666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.130:58277)
0929 20:32:23.726618 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:23.726663 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:23.728639 (+ 1976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:23.729678 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:23.729688 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:23.730306 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:32:23.731121 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:23.731145 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:23.732612 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:23.732617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:23.734557 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:23.734576 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:23.735729 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:23.735735 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:23.736110 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:23.736907 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:32:23.736934 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:32:23.739701 (+ 2767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:32:23.744611 (+ 4910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:23.744626 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:23.744653 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:23.745510 (+ 857us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:23.745984 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:23.745993 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:23.746000 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:23.746306 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:23.747009 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:32:23.747021 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:32:23.747586 (+ 565us) client_negotiation.cc:769] Sending connection context
0929 20:32:23.747987 (+ 401us) client_negotiation.cc:240] Negotiation successful
0929 20:32:23.748708 (+ 721us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":452,"threads_started":1}
I20260929 20:32:23.752261 11403 heartbeater.cc:347] Connected to a master server at 127.28.244.190:37845
I20260929 20:32:23.752712 11403 heartbeater.cc:464] Registering TS with master...
I20260929 20:32:23.753558 11403 heartbeater.cc:511] Master 127.28.244.190:37845 requested a full tablet report, sending...
I20260929 20:32:23.755353 11078 ts_manager.cc:194] Registered new tserver with Master: 8312b6a9e28542de82dc41db65e09a35 (127.28.244.130:40679)
I20260929 20:32:23.756342 11078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.130@KRBTEST.COM'} at 127.28.244.130:58277
WARNING: no policy specified for kudu/127.28.244.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab.
Entry for principal kudu/127.28.244.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab.
I20260929 20:32:23.796933 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.131:0
--local_ip_for_outbound_sockets=127.28.244.131
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:37845
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:34391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:32:24.137108 11267 heartbeater.cc:503] Master 127.28.244.190:37845 was elected leader, sending a full tablet report...
W20260929 20:32:24.217011 11409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:32:24.217355 11409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:32:24.217629 11409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:32:24.227756 11409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:32:24.227977 11409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:32:24.228192 11409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.131
I20260929 20:32:24.238996 11409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:34391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:37845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.131
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:32:24.240514 11409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:32:24.241930 11409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:32:24.258354 11414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:24.258325 11415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:24.258677 11417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:32:24.259447 11409 server_base.cc:1034] running on GCE node
I20260929 20:32:24.260442 11409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:32:24.261577 11409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:32:24.262866 11409 hybrid_clock.cc:648] HybridClock initialized: now 1790713944262803 us; error 53 us; skew 500 ppm
I20260929 20:32:24.263402 11409 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:32:24 dist-test-slave-p29l krb5kdc[11032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713944, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:32:24.267283 11409 init.cc:377] Logged in from keytab as kudu/127.28.244.131@KRBTEST.COM (short username kudu)
I20260929 20:32:24.268992 11409 webserver.cc:460] Webserver started at http://127.28.244.131:41021/ using document root <none> and password file <none>
I20260929 20:32:24.269783 11409 fs_manager.cc:359] Metadata directory not provided
I20260929 20:32:24.269943 11409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:32:24.270269 11409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:32:24.273737 11409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/data/instance:
uuid: "951e78d229d64b709e1b40963e492495"
format_stamp: "Formatted at 2026-09-29 20:32:24 on dist-test-slave-p29l"
server_key: "c450730f0350d3602a4858ebcfe55ff4"
server_key_iv: "f4e6bb5362bdbd100935e6143ad33dca"
server_key_version: "encryptionkey@0"
I20260929 20:32:24.274776 11409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance:
uuid: "951e78d229d64b709e1b40963e492495"
format_stamp: "Formatted at 2026-09-29 20:32:24 on dist-test-slave-p29l"
server_key: "c450730f0350d3602a4858ebcfe55ff4"
server_key_iv: "f4e6bb5362bdbd100935e6143ad33dca"
server_key_version: "encryptionkey@0"
I20260929 20:32:24.281028 11409 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 20:32:24.285926 11424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:24.287714 11409 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260929 20:32:24.287976 11409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/wal
uuid: "951e78d229d64b709e1b40963e492495"
format_stamp: "Formatted at 2026-09-29 20:32:24 on dist-test-slave-p29l"
server_key: "c450730f0350d3602a4858ebcfe55ff4"
server_key_iv: "f4e6bb5362bdbd100935e6143ad33dca"
server_key_version: "encryptionkey@0"
I20260929 20:32:24.288301 11409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:32:24.321592 11409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:32:24.326589 11409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:32:24.326927 11409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:32:24.328126 11409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:32:24.329835 11409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:32:24.329949 11409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:24.330106 11409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:32:24.330186 11409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:24.370042 11409 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.131:40541
I20260929 20:32:24.370281 11538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.131:40541 every 8 connection(s)
I20260929 20:32:24.372143 11409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
I20260929 20:32:24.380264 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 11409
I20260929 20:32:24.380472 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance
I20260929 20:32:24.381040 29650 external_mini_cluster.cc:1428] Setting key ee7a5925297af94a006272c1e5cf75de
Sep 29 20:32:24 dist-test-slave-p29l krb5kdc[11032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713944, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:32:24.396523 11541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:24.375609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:43887 (local address 127.28.244.190:37845)
0929 20:32:24.376224 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:24.376228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:24.377540 (+ 1312us) server_negotiation.cc:407] Connection header received
0929 20:32:24.379521 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:24.379527 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:24.379640 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:24.379766 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:24.381682 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:24.382659 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:24.385001 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:24.385408 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:24.385556 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:24.389738 (+ 4182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:24.389745 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:24.389748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:24.389788 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:24.392940 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:24.393847 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:24.393853 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:24.393862 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:24.394033 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:24.394516 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:24.394521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:24.394524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:24.394846 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:24.395018 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:24.395787 (+ 769us) server_negotiation.cc:299] Negotiation successful
0929 20:32:24.396098 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":339,"threads_started":1}
I20260929 20:32:24.397680 11542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:24.376328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.131:43887)
0929 20:32:24.377339 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:24.377381 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:24.379200 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:24.380083 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:24.380094 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:24.380755 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:32:24.381474 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:24.381497 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:24.382848 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:24.382852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:24.384813 (+ 1961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:24.384829 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:24.385662 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:24.385671 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:24.385941 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:24.386931 (+ 990us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:32:24.386957 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:32:24.389439 (+ 2482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:32:24.393140 (+ 3701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:24.393152 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:24.393177 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:24.393690 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:24.394161 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:24.394166 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:24.394170 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:24.394388 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:24.394982 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:32:24.394991 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:32:24.395394 (+ 403us) client_negotiation.cc:769] Sending connection context
0929 20:32:24.395682 (+ 288us) client_negotiation.cc:240] Negotiation successful
0929 20:32:24.396306 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"thread_start_us":504,"threads_started":1}
I20260929 20:32:24.399523 11539 heartbeater.cc:347] Connected to a master server at 127.28.244.190:37845
I20260929 20:32:24.399955 11539 heartbeater.cc:464] Registering TS with master...
I20260929 20:32:24.400774 11539 heartbeater.cc:511] Master 127.28.244.190:37845 requested a full tablet report, sending...
I20260929 20:32:24.402554 11078 ts_manager.cc:194] Registered new tserver with Master: 951e78d229d64b709e1b40963e492495 (127.28.244.131:40541)
I20260929 20:32:24.403479 11078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.131@KRBTEST.COM'} at 127.28.244.131:43887
I20260929 20:32:24.410531 29650 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:32:24.431389 11541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:24.415583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45838 (local address 127.28.244.190:37845)
0929 20:32:24.415940 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:24.415949 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:24.416244 (+ 295us) server_negotiation.cc:407] Connection header received
0929 20:32:24.416485 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:24.416489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:24.416559 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:24.416680 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:24.418393 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:24.419263 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:24.421049 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:24.421537 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:24.421700 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:24.424678 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:24.424684 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:24.424688 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:24.424729 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:24.428005 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:24.428934 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:24.428940 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:24.428943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:24.429095 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:24.429593 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:24.429598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:24.429601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:24.430053 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:24.430345 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:24.430742 (+ 397us) server_negotiation.cc:299] Negotiation successful
0929 20:32:24.431015 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260929 20:32:24.436705 11078 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:32:24.444408 11078 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:32:24.485848 11552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:24.460361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:33535 (local address 127.0.0.1:43904)
0929 20:32:24.461901 (+ 1540us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:24.461944 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:24.462136 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:24.477264 (+ 15128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:24.477273 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:24.477316 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:32:24.477802 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:24.477820 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:24.479905 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:24.479912 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:24.482571 (+ 2659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:24.482589 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:24.484638 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:24.484644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:24.484880 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:24.484956 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 20:32:24.485121 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 20:32:24.485341 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1277,"mutex_wait_us":4,"thread_start_us":1088,"threads_started":1}
I20260929 20:32:24.485901 11553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:24.460638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43904 (local address 127.28.244.129:33535)
0929 20:32:24.476312 (+ 15674us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:24.476324 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:24.476368 (+ 44us) server_negotiation.cc:407] Connection header received
0929 20:32:24.476510 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:24.476525 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:24.476898 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:24.477158 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:32:24.478274 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:24.479687 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:24.482792 (+ 3105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:24.484372 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:24.484622 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:24.484829 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:24.485353 (+ 524us) server_negotiation.cc:299] Negotiation successful
0929 20:32:24.485541 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15437,"thread_start_us":15960,"threads_started":1}
I20260929 20:32:24.486449 11555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:24.461955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:40541 (local address 127.0.0.1:52698)
0929 20:32:24.476308 (+ 14353us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:24.476330 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:24.476489 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:24.478021 (+ 1532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:24.478030 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:24.478073 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:32:24.478735 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:24.478754 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:24.480845 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:24.480851 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:24.482528 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:24.482543 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:24.485034 (+ 2491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:24.485040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:24.485266 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:24.485296 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 20:32:24.485380 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 20:32:24.485509 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14083,"mutex_wait_us":71,"thread_start_us":16996,"threads_started":1}
I20260929 20:32:24.486816 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:24.462819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52698 (local address 127.28.244.131:40541)
0929 20:32:24.477282 (+ 14463us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:24.477291 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:24.477324 (+ 33us) server_negotiation.cc:407] Connection header received
0929 20:32:24.477482 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:24.477490 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:24.477755 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:24.477996 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:32:24.479007 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:24.480672 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:24.482720 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:24.484803 (+ 2083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:24.485021 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:24.485217 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:24.485881 (+ 664us) server_negotiation.cc:299] Negotiation successful
0929 20:32:24.486106 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14250,"thread_start_us":14105,"threads_started":1}
I20260929 20:32:24.488807 11556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:24.462683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60986 (local address 127.28.244.130:40679)
0929 20:32:24.475255 (+ 12572us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:24.475270 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:24.479408 (+ 4138us) server_negotiation.cc:407] Connection header received
0929 20:32:24.479734 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:24.479743 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:24.480055 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:24.480235 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:32:24.481191 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:24.482706 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:24.485327 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:24.487304 (+ 1977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:24.487526 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:24.488029 (+ 503us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:24.488245 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 20:32:24.488434 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12274,"thread_start_us":12596,"threads_started":1}
I20260929 20:32:24.489467 11554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:24.462402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:40679 (local address 127.0.0.1:60986)
0929 20:32:24.479270 (+ 16868us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:24.479289 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:24.479522 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:24.480493 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:24.480497 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:24.480524 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:32:24.480996 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:24.481007 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:24.482852 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:24.482857 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:24.485112 (+ 2255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:24.485128 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:24.487766 (+ 2638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:24.487781 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:24.488012 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:24.488046 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 20:32:24.488895 (+ 849us) client_negotiation.cc:240] Negotiation successful
0929 20:32:24.489048 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16731,"mutex_wait_us":26,"thread_start_us":13362,"threads_started":1}
I20260929 20:32:24.491109 11202 tablet_service.cc:1467] Processing CreateTablet for tablet 8011286b473e4f598bc444a2692aed50 (DEFAULT_TABLE table=test-table [id=d460186287e14c8382ac1b96351f10c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:32:24.491102 11473 tablet_service.cc:1467] Processing CreateTablet for tablet 8011286b473e4f598bc444a2692aed50 (DEFAULT_TABLE table=test-table [id=d460186287e14c8382ac1b96351f10c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:32:24.492852 11473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8011286b473e4f598bc444a2692aed50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:32:24.492856 11202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8011286b473e4f598bc444a2692aed50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:32:24.493163 11336 tablet_service.cc:1467] Processing CreateTablet for tablet 8011286b473e4f598bc444a2692aed50 (DEFAULT_TABLE table=test-table [id=d460186287e14c8382ac1b96351f10c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:32:24.495308 11336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8011286b473e4f598bc444a2692aed50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:32:24.509016 11558 tablet_bootstrap.cc:492] T 8011286b473e4f598bc444a2692aed50 P 8312b6a9e28542de82dc41db65e09a35: Bootstrap starting.
I20260929 20:32:24.510090 11559 tablet_bootstrap.cc:492] T 8011286b473e4f598bc444a2692aed50 P e1aa120e44bb45e68986e21a8565480e: Bootstrap starting.
I20260929 20:32:24.512411 11560 tablet_bootstrap.cc:492] T 8011286b473e4f598bc444a2692aed50 P 951e78d229d64b709e1b40963e492495: Bootstrap starting.
I20260929 20:32:24.514328 11559 tablet_bootstrap.cc:654] T 8011286b473e4f598bc444a2692aed50 P e1aa120e44bb45e68986e21a8565480e: Neither blocks nor log segments found. Creating new log.
I20260929 20:32:24.514328 11558 tablet_bootstrap.cc:654] T 8011286b473e4f598bc444a2692aed50 P 8312b6a9e28542de82dc41db65e09a35: Neither blocks nor log segments found. Creating new log.
I20260929 20:32:24.515764 11559 log.cc:824] T 8011286b473e4f598bc444a2692aed50 P e1aa120e44bb45e68986e21a8565480e: Log is configured to *not* fsync() on all Append() calls
I20260929 20:32:24.515764 11558 log.cc:824] T 8011286b473e4f598bc444a2692aed50 P 8312b6a9e28542de82dc41db65e09a35: Log is configured to *not* fsync() on all Append() calls
I20260929 20:32:24.516182 11560 tablet_bootstrap.cc:654] T 8011286b473e4f598bc444a2692aed50 P 951e78d229d64b709e1b40963e492495: Neither blocks nor log segments found. Creating new log.
I20260929 20:32:24.517735 11560 log.cc:824] T 8011286b473e4f598bc444a2692aed50 P 951e78d229d64b709e1b40963e492495: Log is configured to *not* fsync() on all Append() calls
I20260929 20:32:24.520674 11558 tablet_bootstrap.cc:492] T 8011286b473e4f598bc444a2692aed50 P 8312b6a9e28542de82dc41db65e09a35: No bootstrap required, opened a new log
I20260929 20:32:24.521040 11558 ts_tablet_manager.cc:1397] T 8011286b473e4f598bc444a2692aed50 P 8312b6a9e28542de82dc41db65e09a35: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.002s
I20260929 20:32:24.521100 11559 tablet_bootstrap.cc:492] T 8011286b473e4f598bc444a2692aed50 P e1aa120e44bb45e68986e21a8565480e: No bootstrap required, opened a new log
I20260929 20:32:24.521391 11559 ts_tablet_manager.cc:1397] T 8011286b473e4f598bc444a2692aed50 P e1aa120e44bb45e68986e21a8565480e: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260929 20:32:24.521870 11560 tablet_bootstrap.cc:492] T 8011286b473e4f598bc444a2692aed50 P 951e78d229d64b709e1b40963e492495: No bootstrap required, opened a new log
I20260929 20:32:24.522145 11560 ts_tablet_manager.cc:1397] T 8011286b473e4f598bc444a2692aed50 P 951e78d229d64b709e1b40963e492495: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 20:32:24.530342 11559 raft_consensus.cc:348] T 8011286b473e4f598bc444a2692aed50 P e1aa120e44bb45e68986e21a8565480e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1aa120e44bb45e68986e21a8565480e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 33535 } } peers { permanent_uuid: "8312b6a9e28542de82dc41db65e09a35" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40679 } } peers { permanent_uuid: "951e78d229d64b709e1b40963e492495" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 40541 } }
I20260929 20:32:24.530892 11559 raft_consensus.cc:374] T 8011286b473e4f598bc444a2692aed50 P e1aa120e44bb45e68986e21a8565480e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:32:24.531026 11559 raft_consensus.cc:729] T 8011286b473e4f598bc444a2692aed50 P e1aa120e44bb45e68986e21a8565480e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1aa120e44bb45e68986e21a8565480e, State: Initialized, Role: FOLLOWER
I20260929 20:32:24.531759 11559 consensus_queue.cc:260] T 8011286b473e4f598bc444a2692aed50 P e1aa120e44bb45e68986e21a8565480e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1aa120e44bb45e68986e21a8565480e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 33535 } } peers { permanent_uuid: "8312b6a9e28542de82dc41db65e09a35" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40679 } } peers { permanent_uuid: "951e78d229d64b709e1b40963e492495" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 40541 } }
I20260929 20:32:24.533419 11558 raft_consensus.cc:348] T 8011286b473e4f598bc444a2692aed50 P 8312b6a9e28542de82dc41db65e09a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1aa120e44bb45e68986e21a8565480e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 33535 } } peers { permanent_uuid: "8312b6a9e28542de82dc41db65e09a35" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40679 } } peers { permanent_uuid: "951e78d229d64b709e1b40963e492495" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 40541 } }
I20260929 20:32:24.533820 11559 ts_tablet_manager.cc:1428] T 8011286b473e4f598bc444a2692aed50 P e1aa120e44bb45e68986e21a8565480e: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260929 20:32:24.534039 11558 raft_consensus.cc:374] T 8011286b473e4f598bc444a2692aed50 P 8312b6a9e28542de82dc41db65e09a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:32:24.534194 11558 raft_consensus.cc:729] T 8011286b473e4f598bc444a2692aed50 P 8312b6a9e28542de82dc41db65e09a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8312b6a9e28542de82dc41db65e09a35, State: Initialized, Role: FOLLOWER
I20260929 20:32:24.534004 11560 raft_consensus.cc:348] T 8011286b473e4f598bc444a2692aed50 P 951e78d229d64b709e1b40963e492495 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1aa120e44bb45e68986e21a8565480e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 33535 } } peers { permanent_uuid: "8312b6a9e28542de82dc41db65e09a35" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40679 } } peers { permanent_uuid: "951e78d229d64b709e1b40963e492495" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 40541 } }
I20260929 20:32:24.534484 11560 raft_consensus.cc:374] T 8011286b473e4f598bc444a2692aed50 P 951e78d229d64b709e1b40963e492495 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:32:24.534626 11560 raft_consensus.cc:729] T 8011286b473e4f598bc444a2692aed50 P 951e78d229d64b709e1b40963e492495 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951e78d229d64b709e1b40963e492495, State: Initialized, Role: FOLLOWER
I20260929 20:32:24.534974 11558 consensus_queue.cc:260] T 8011286b473e4f598bc444a2692aed50 P 8312b6a9e28542de82dc41db65e09a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1aa120e44bb45e68986e21a8565480e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 33535 } } peers { permanent_uuid: "8312b6a9e28542de82dc41db65e09a35" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40679 } } peers { permanent_uuid: "951e78d229d64b709e1b40963e492495" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 40541 } }
I20260929 20:32:24.535233 11560 consensus_queue.cc:260] T 8011286b473e4f598bc444a2692aed50 P 951e78d229d64b709e1b40963e492495 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1aa120e44bb45e68986e21a8565480e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 33535 } } peers { permanent_uuid: "8312b6a9e28542de82dc41db65e09a35" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40679 } } peers { permanent_uuid: "951e78d229d64b709e1b40963e492495" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 40541 } }
I20260929 20:32:24.536528 11539 heartbeater.cc:503] Master 127.28.244.190:37845 was elected leader, sending a full tablet report...
I20260929 20:32:24.537619 11560 ts_tablet_manager.cc:1428] T 8011286b473e4f598bc444a2692aed50 P 951e78d229d64b709e1b40963e492495: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260929 20:32:24.539906 11403 heartbeater.cc:503] Master 127.28.244.190:37845 was elected leader, sending a full tablet report...
I20260929 20:32:24.540916 11558 ts_tablet_manager.cc:1428] T 8011286b473e4f598bc444a2692aed50 P 8312b6a9e28542de82dc41db65e09a35: Time spent starting tablet: real 0.020s user 0.015s sys 0.004s
I20260929 20:32:24.564615 11564 raft_consensus.cc:482] T 8011286b473e4f598bc444a2692aed50 P e1aa120e44bb45e68986e21a8565480e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:32:24.564996 11564 raft_consensus.cc:504] T 8011286b473e4f598bc444a2692aed50 P e1aa120e44bb45e68986e21a8565480e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1aa120e44bb45e68986e21a8565480e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 33535 } } peers { permanent_uuid: "8312b6a9e28542de82dc41db65e09a35" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40679 } } peers { permanent_uuid: "951e78d229d64b709e1b40963e492495" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 40541 } }
I20260929 20:32:24.567085 11564 leader_election.cc:288] T 8011286b473e4f598bc444a2692aed50 P e1aa120e44bb45e68986e21a8565480e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8312b6a9e28542de82dc41db65e09a35 (127.28.244.130:40679), 951e78d229d64b709e1b40963e492495 (127.28.244.131:40541)
I20260929 20:32:24.576207 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:24.567554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:48573 (local address 127.28.244.131:40541)
0929 20:32:24.567799 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:24.567828 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:24.568883 (+ 1055us) server_negotiation.cc:407] Connection header received
0929 20:32:24.569102 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:24.569106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:24.569169 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:24.569272 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:32:24.569927 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:24.570652 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:24.573214 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:24.574501 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:24.574678 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:24.574762 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:24.575720 (+ 958us) server_negotiation.cc:299] Negotiation successful
0929 20:32:24.575909 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 20:32:24.576207 11556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:24.567763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:60109 (local address 127.28.244.130:40679)
0929 20:32:24.567991 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:24.568000 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:24.568739 (+ 739us) server_negotiation.cc:407] Connection header received
0929 20:32:24.568968 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:24.568973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:24.569033 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:24.569135 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:32:24.569776 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:24.570522 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:24.573214 (+ 2692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:24.574494 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:24.574678 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:24.574770 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:24.575721 (+ 951us) server_negotiation.cc:299] Negotiation successful
0929 20:32:24.575902 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:32:24.576459 11567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:24.567568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:40679 (local address 127.28.244.129:60109)
0929 20:32:24.568560 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:24.568613 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:24.568808 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:24.569167 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:24.569172 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:24.569198 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:32:24.569607 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:24.569618 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:24.570660 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:24.570668 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:24.572994 (+ 2326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:24.573014 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:24.574906 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:24.574910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:24.575409 (+ 499us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:24.575516 (+ 107us) client_negotiation.cc:769] Sending connection context
0929 20:32:24.575652 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 20:32:24.575875 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"mutex_wait_us":270,"thread_start_us":556,"threads_started":1}
I20260929 20:32:24.576483 11568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:24.567617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:40541 (local address 127.28.244.129:48573)
0929 20:32:24.568760 (+ 1143us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:24.568775 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:24.568946 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:24.569298 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:24.569302 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:24.569347 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:32:24.569745 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:24.569755 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:24.570809 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:24.570815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:24.572994 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:24.573014 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:24.574894 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:24.574903 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:24.575409 (+ 506us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:24.575522 (+ 113us) client_negotiation.cc:769] Sending connection context
0929 20:32:24.575648 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 20:32:24.575874 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1042,"mutex_wait_us":442,"thread_start_us":794,"threads_started":1}
I20260929 20:32:24.577629 11357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8011286b473e4f598bc444a2692aed50" candidate_uuid: "e1aa120e44bb45e68986e21a8565480e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8312b6a9e28542de82dc41db65e09a35" is_pre_election: true
I20260929 20:32:24.578171 11357 raft_consensus.cc:2454] T 8011286b473e4f598bc444a2692aed50 P 8312b6a9e28542de82dc41db65e09a35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1aa120e44bb45e68986e21a8565480e in term 0.
I20260929 20:32:24.578354 11493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8011286b473e4f598bc444a2692aed50" candidate_uuid: "e1aa120e44bb45e68986e21a8565480e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "951e78d229d64b709e1b40963e492495" is_pre_election: true
I20260929 20:32:24.578886 11493 raft_consensus.cc:2454] T 8011286b473e4f598bc444a2692aed50 P 951e78d229d64b709e1b40963e492495 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1aa120e44bb45e68986e21a8565480e in term 0.
I20260929 20:32:24.578902 11156 leader_election.cc:302] T 8011286b473e4f598bc444a2692aed50 P e1aa120e44bb45e68986e21a8565480e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8312b6a9e28542de82dc41db65e09a35, e1aa120e44bb45e68986e21a8565480e; no voters:
I20260929 20:32:24.579298 11564 raft_consensus.cc:2781] T 8011286b473e4f598bc444a2692aed50 P e1aa120e44bb45e68986e21a8565480e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:32:24.579500 11564 raft_consensus.cc:482] T 8011286b473e4f598bc444a2692aed50 P e1aa120e44bb45e68986e21a8565480e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:32:24.579641 11564 raft_consensus.cc:3037] T 8011286b473e4f598bc444a2692aed50 P e1aa120e44bb45e68986e21a8565480e [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:32:24.582274 11564 raft_consensus.cc:504] T 8011286b473e4f598bc444a2692aed50 P e1aa120e44bb45e68986e21a8565480e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1aa120e44bb45e68986e21a8565480e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 33535 } } peers { permanent_uuid: "8312b6a9e28542de82dc41db65e09a35" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40679 } } peers { permanent_uuid: "951e78d229d64b709e1b40963e492495" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 40541 } }
I20260929 20:32:24.583021 11564 leader_election.cc:288] T 8011286b473e4f598bc444a2692aed50 P e1aa120e44bb45e68986e21a8565480e [CANDIDATE]: Term 1 election: Requested vote from peers 8312b6a9e28542de82dc41db65e09a35 (127.28.244.130:40679), 951e78d229d64b709e1b40963e492495 (127.28.244.131:40541)
I20260929 20:32:24.583616 11357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8011286b473e4f598bc444a2692aed50" candidate_uuid: "e1aa120e44bb45e68986e21a8565480e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8312b6a9e28542de82dc41db65e09a35"
I20260929 20:32:24.583679 11493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8011286b473e4f598bc444a2692aed50" candidate_uuid: "e1aa120e44bb45e68986e21a8565480e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "951e78d229d64b709e1b40963e492495"
I20260929 20:32:24.583930 11357 raft_consensus.cc:3037] T 8011286b473e4f598bc444a2692aed50 P 8312b6a9e28542de82dc41db65e09a35 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:32:24.583969 11493 raft_consensus.cc:3037] T 8011286b473e4f598bc444a2692aed50 P 951e78d229d64b709e1b40963e492495 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:32:24.586330 11357 raft_consensus.cc:2454] T 8011286b473e4f598bc444a2692aed50 P 8312b6a9e28542de82dc41db65e09a35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1aa120e44bb45e68986e21a8565480e in term 1.
I20260929 20:32:24.586354 11493 raft_consensus.cc:2454] T 8011286b473e4f598bc444a2692aed50 P 951e78d229d64b709e1b40963e492495 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1aa120e44bb45e68986e21a8565480e in term 1.
I20260929 20:32:24.586923 11155 leader_election.cc:302] T 8011286b473e4f598bc444a2692aed50 P e1aa120e44bb45e68986e21a8565480e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 951e78d229d64b709e1b40963e492495, e1aa120e44bb45e68986e21a8565480e; no voters:
I20260929 20:32:24.587468 11564 raft_consensus.cc:2781] T 8011286b473e4f598bc444a2692aed50 P e1aa120e44bb45e68986e21a8565480e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:32:24.588052 11564 raft_consensus.cc:686] T 8011286b473e4f598bc444a2692aed50 P e1aa120e44bb45e68986e21a8565480e [term 1 LEADER]: Becoming Leader. State: Replica: e1aa120e44bb45e68986e21a8565480e, State: Running, Role: LEADER
I20260929 20:32:24.588637 11564 consensus_queue.cc:237] T 8011286b473e4f598bc444a2692aed50 P e1aa120e44bb45e68986e21a8565480e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1aa120e44bb45e68986e21a8565480e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 33535 } } peers { permanent_uuid: "8312b6a9e28542de82dc41db65e09a35" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40679 } } peers { permanent_uuid: "951e78d229d64b709e1b40963e492495" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 40541 } }
W20260929 20:32:24.593933 11268 tablet.cc:2369] T 8011286b473e4f598bc444a2692aed50 P e1aa120e44bb45e68986e21a8565480e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:32:24.594660 11078 catalog_manager.cc:5495] T 8011286b473e4f598bc444a2692aed50 P e1aa120e44bb45e68986e21a8565480e reported cstate change: term changed from 0 to 1, leader changed from <none> to e1aa120e44bb45e68986e21a8565480e (127.28.244.129). New cstate: current_term: 1 leader_uuid: "e1aa120e44bb45e68986e21a8565480e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1aa120e44bb45e68986e21a8565480e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 33535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8312b6a9e28542de82dc41db65e09a35" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "951e78d229d64b709e1b40963e492495" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 40541 } health_report { overall_health: UNKNOWN } } }
I20260929 20:32:24.619128 11553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:24.613026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43914 (local address 127.28.244.129:33535)
0929 20:32:24.613290 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:24.613295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:24.613388 (+ 93us) server_negotiation.cc:407] Connection header received
0929 20:32:24.613570 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:24.613580 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:24.613669 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:24.613804 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:32:24.614195 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:24.615008 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:24.616642 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:24.617069 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:24.617187 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:24.617915 (+ 728us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:32:24.618484 (+ 569us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:32:24.618630 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:24.618735 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 20:32:24.618855 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260929 20:32:24.624362 11540 tablet.cc:2369] T 8011286b473e4f598bc444a2692aed50 P 951e78d229d64b709e1b40963e492495: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:32:24.628494 11357 raft_consensus.cc:1261] T 8011286b473e4f598bc444a2692aed50 P 8312b6a9e28542de82dc41db65e09a35 [term 1 FOLLOWER]: Refusing update from remote peer e1aa120e44bb45e68986e21a8565480e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:32:24.628494 11493 raft_consensus.cc:1261] T 8011286b473e4f598bc444a2692aed50 P 951e78d229d64b709e1b40963e492495 [term 1 FOLLOWER]: Refusing update from remote peer e1aa120e44bb45e68986e21a8565480e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:32:24.629575 11569 consensus_queue.cc:1035] T 8011286b473e4f598bc444a2692aed50 P e1aa120e44bb45e68986e21a8565480e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8312b6a9e28542de82dc41db65e09a35" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:32:24.629989 11564 consensus_queue.cc:1035] T 8011286b473e4f598bc444a2692aed50 P e1aa120e44bb45e68986e21a8565480e [LEADER]: Connected to new peer: Peer: permanent_uuid: "951e78d229d64b709e1b40963e492495" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 40541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:32:24.640331 11572 mvcc.cc:205] Tried to move back new op lower bound from 7334764317187584000 to 7334764317039972352. Current Snapshot: MvccSnapshot[applied={T|T < 7334764317187584000}]
I20260929 20:32:24.642112 11573 mvcc.cc:205] Tried to move back new op lower bound from 7334764317187584000 to 7334764317039972352. Current Snapshot: MvccSnapshot[applied={T|T < 7334764317187584000}]
I20260929 20:32:24.657061 11078 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:32:24.657368 11078 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:32:24.662365 11078 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a: Sending DeleteTablet for 3 replicas of tablet 8011286b473e4f598bc444a2692aed50
I20260929 20:32:24.664103 11202 tablet_service.cc:1514] Processing DeleteTablet for tablet 8011286b473e4f598bc444a2692aed50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:32:24 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:43904
I20260929 20:32:24.664453 11473 tablet_service.cc:1514] Processing DeleteTablet for tablet 8011286b473e4f598bc444a2692aed50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:32:24 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:52698
I20260929 20:32:24.664552 11472 tablet_service.cc:1514] Processing DeleteTablet for tablet 8011286b473e4f598bc444a2692aed50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:32:24 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:52698
I20260929 20:32:24.663765 11336 tablet_service.cc:1514] Processing DeleteTablet for tablet 8011286b473e4f598bc444a2692aed50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:32:24 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:60986
I20260929 20:32:24.665359 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 11139
I20260929 20:32:24.674268 11582 tablet_replica.cc:330] T 8011286b473e4f598bc444a2692aed50 P e1aa120e44bb45e68986e21a8565480e: stopping tablet replica
I20260929 20:32:24.674569 11583 tablet_replica.cc:330] T 8011286b473e4f598bc444a2692aed50 P 951e78d229d64b709e1b40963e492495: stopping tablet replica
I20260929 20:32:24.674818 11582 raft_consensus.cc:2229] T 8011286b473e4f598bc444a2692aed50 P e1aa120e44bb45e68986e21a8565480e [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:32:24.675555 11583 raft_consensus.cc:2229] T 8011286b473e4f598bc444a2692aed50 P 951e78d229d64b709e1b40963e492495 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:32:24.675769 11582 raft_consensus.cc:2258] T 8011286b473e4f598bc444a2692aed50 P e1aa120e44bb45e68986e21a8565480e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:32:24.676257 11583 raft_consensus.cc:2258] T 8011286b473e4f598bc444a2692aed50 P 951e78d229d64b709e1b40963e492495 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:32:24.678556 11583 ts_tablet_manager.cc:1905] T 8011286b473e4f598bc444a2692aed50 P 951e78d229d64b709e1b40963e492495: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:32:24.680810 11582 ts_tablet_manager.cc:1905] T 8011286b473e4f598bc444a2692aed50 P e1aa120e44bb45e68986e21a8565480e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:32:24.682060 11584 tablet_replica.cc:330] T 8011286b473e4f598bc444a2692aed50 P 8312b6a9e28542de82dc41db65e09a35: stopping tablet replica
I20260929 20:32:24.682530 11584 raft_consensus.cc:2229] T 8011286b473e4f598bc444a2692aed50 P 8312b6a9e28542de82dc41db65e09a35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:32:24.683071 11584 raft_consensus.cc:2258] T 8011286b473e4f598bc444a2692aed50 P 8312b6a9e28542de82dc41db65e09a35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:32:24.686406 11583 ts_tablet_manager.cc:1918] T 8011286b473e4f598bc444a2692aed50 P 951e78d229d64b709e1b40963e492495: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:32:24.686650 11583 log.cc:1196] T 8011286b473e4f598bc444a2692aed50 P 951e78d229d64b709e1b40963e492495: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/wal/wals/8011286b473e4f598bc444a2692aed50
I20260929 20:32:24.687215 11584 ts_tablet_manager.cc:1905] T 8011286b473e4f598bc444a2692aed50 P 8312b6a9e28542de82dc41db65e09a35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:32:24.687402 11583 ts_tablet_manager.cc:1939] T 8011286b473e4f598bc444a2692aed50 P 951e78d229d64b709e1b40963e492495: Deleting consensus metadata
I20260929 20:32:24.689555 11582 ts_tablet_manager.cc:1918] T 8011286b473e4f598bc444a2692aed50 P e1aa120e44bb45e68986e21a8565480e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:32:24.689610 11061 catalog_manager.cc:4841] TS 951e78d229d64b709e1b40963e492495 (127.28.244.131:40541): tablet 8011286b473e4f598bc444a2692aed50 (table test-table [id=d460186287e14c8382ac1b96351f10c3]) successfully deleted
I20260929 20:32:24.689836 11582 log.cc:1196] T 8011286b473e4f598bc444a2692aed50 P e1aa120e44bb45e68986e21a8565480e: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/wal/wals/8011286b473e4f598bc444a2692aed50
I20260929 20:32:24.690626 11582 ts_tablet_manager.cc:1939] T 8011286b473e4f598bc444a2692aed50 P e1aa120e44bb45e68986e21a8565480e: Deleting consensus metadata
W20260929 20:32:24.690909 11061 catalog_manager.cc:4808] TS 951e78d229d64b709e1b40963e492495 (127.28.244.131:40541): delete failed for tablet 8011286b473e4f598bc444a2692aed50 because the tablet was not found. No further retry: Not found: Tablet not found: 8011286b473e4f598bc444a2692aed50
I20260929 20:32:24.692791 11062 catalog_manager.cc:4841] TS e1aa120e44bb45e68986e21a8565480e (127.28.244.129:33535): tablet 8011286b473e4f598bc444a2692aed50 (table test-table [id=d460186287e14c8382ac1b96351f10c3]) successfully deleted
Sep 29 20:32:24 dist-test-slave-p29l krb5kdc[11032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.129@KRBTEST.COM
I20260929 20:32:24.696161 11584 ts_tablet_manager.cc:1918] T 8011286b473e4f598bc444a2692aed50 P 8312b6a9e28542de82dc41db65e09a35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:32:24.696372 11584 log.cc:1196] T 8011286b473e4f598bc444a2692aed50 P 8312b6a9e28542de82dc41db65e09a35: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/wal/wals/8011286b473e4f598bc444a2692aed50
I20260929 20:32:24.696875 11584 ts_tablet_manager.cc:1939] T 8011286b473e4f598bc444a2692aed50 P 8312b6a9e28542de82dc41db65e09a35: Deleting consensus metadata
I20260929 20:32:24.699229 11062 catalog_manager.cc:4841] TS 8312b6a9e28542de82dc41db65e09a35 (127.28.244.130:40679): tablet 8011286b473e4f598bc444a2692aed50 (table test-table [id=d460186287e14c8382ac1b96351f10c3]) successfully deleted
I20260929 20:32:24.704897 11553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:24.666816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43920 (local address 127.28.244.129:33535)
0929 20:32:24.667014 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:24.667021 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:24.673705 (+ 6684us) server_negotiation.cc:407] Connection header received
0929 20:32:24.674523 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:24.674528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:24.674615 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:24.674700 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:24.678027 (+ 3327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:24.679470 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:24.683528 (+ 4058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:24.684247 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:24.684359 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:24.697347 (+ 12988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:24.697368 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:24.697388 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:24.697462 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:24.701345 (+ 3883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:24.702466 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:24.702476 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:24.702479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:24.702643 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:24.703129 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:24.703133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:24.703135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:24.703543 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:24.703927 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:24.704264 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 20:32:24.704484 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:32:24.705811 11262 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:32:24.813354 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 11139
I20260929 20:32:24.834764 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 11273
Sep 29 20:32:24 dist-test-slave-p29l krb5kdc[11032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.130@KRBTEST.COM
I20260929 20:32:24.859018 11556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:24.835628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32768 (local address 127.28.244.130:40679)
0929 20:32:24.835906 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:24.835910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:24.835939 (+ 29us) server_negotiation.cc:407] Connection header received
0929 20:32:24.836053 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:24.836057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:24.836125 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:24.836212 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:24.837625 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:24.838658 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:24.840514 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:24.841008 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:24.841150 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:24.850551 (+ 9401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:24.850571 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:24.850589 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:24.850650 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:24.855007 (+ 4357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:24.856286 (+ 1279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:24.856297 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:24.856300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:24.856468 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:24.856953 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:24.856957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:24.856959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:24.857446 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:24.857723 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:24.858305 (+ 582us) server_negotiation.cc:299] Negotiation successful
0929 20:32:24.858586 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 20:32:24.859906 11398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:32:24.958859 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 11273
I20260929 20:32:24.977784 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 11409
Sep 29 20:32:24 dist-test-slave-p29l krb5kdc[11032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.131@KRBTEST.COM
I20260929 20:32:24.996645 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:24.978787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52710 (local address 127.28.244.131:40541)
0929 20:32:24.978960 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:24.978965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:24.978988 (+ 23us) server_negotiation.cc:407] Connection header received
0929 20:32:24.979203 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:24.979216 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:24.979294 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:24.979392 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:24.980915 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:24.981685 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:24.983442 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:24.984222 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:24.984369 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:24.989231 (+ 4862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:24.989254 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:24.989279 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:24.989327 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:24.992601 (+ 3274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:24.993597 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:24.993605 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:24.993607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:24.993835 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:24.994313 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:24.994318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:24.994320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:24.994785 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:24.995154 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:24.995486 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 20:32:24.995684 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 20:32:24.998003 11534 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:32:25.096201 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 11409
I20260929 20:32:25.115651 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 11045
I20260929 20:32:25.118068 11108 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:32:25.253361 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 11045
I20260929 20:32:25.280278 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:37845
--webserver_interface=127.28.244.190
--webserver_port=40263
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:34391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:37845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:32:25.664366 11590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:32:25.664736 11590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:32:25.664906 11590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:32:25.674178 11590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:32:25.674337 11590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:32:25.674388 11590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:32:25.674443 11590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:32:25.674567 11590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:32:25.685554 11590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:34391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:37845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:37845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=40263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:32:25.687060 11590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:32:25.688633 11590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:32:25.716189 11596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:25.716181 11597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:25.716535 11599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:32:25.717075 11590 server_base.cc:1034] running on GCE node
I20260929 20:32:25.718124 11590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:32:25.720185 11590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:32:25.721483 11590 hybrid_clock.cc:648] HybridClock initialized: now 1790713945721424 us; error 67 us; skew 500 ppm
I20260929 20:32:25.722021 11590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:32:25 dist-test-slave-p29l krb5kdc[11032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713945, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:32:25.735349 11590 init.cc:377] Logged in from keytab as kudu/127.28.244.190@KRBTEST.COM (short username kudu)
I20260929 20:32:25.737557 11590 webserver.cc:460] Webserver started at http://127.28.244.190:40263/ using document root <none> and password file <none>
I20260929 20:32:25.738420 11590 fs_manager.cc:359] Metadata directory not provided
I20260929 20:32:25.738559 11590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:32:25.745599 11590 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 20:32:25.749675 11606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:25.752193 11590 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.003s
I20260929 20:32:25.752403 11590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "7419d303241e4b149f92d21d7bc9132a"
format_stamp: "Formatted at 2026-09-29 20:32:22 on dist-test-slave-p29l"
server_key: "4b0c1ac5b33c319c2789b0ab2aad0838"
server_key_iv: "9561148bfb031b03b0d3b191f5ee30e1"
server_key_version: "encryptionkey@0"
I20260929 20:32:25.753648 11590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:32:25.772409 11590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:32:25.777622 11590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:32:25.777887 11590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:32:25.802585 11590 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:37845
I20260929 20:32:25.802598 11657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:37845 every 8 connection(s)
I20260929 20:32:25.804442 11590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:32:25.808260 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 11590
I20260929 20:32:25.809286 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.129:33535
--local_ip_for_outbound_sockets=127.28.244.129
--tserver_master_addrs=127.28.244.190:37845
--webserver_port=45621
--webserver_interface=127.28.244.129
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:34391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:32:25.813911 11658 sys_catalog.cc:263] Verifying existing consensus state
I20260929 20:32:25.819725 11658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a: Bootstrap starting.
I20260929 20:32:25.842080 11658 log.cc:824] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a: Log is configured to *not* fsync() on all Append() calls
I20260929 20:32:25.857439 11658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:32:25.858180 11658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a: Bootstrap complete.
I20260929 20:32:25.872035 11658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7419d303241e4b149f92d21d7bc9132a" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 37845 } }
I20260929 20:32:25.873728 11658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7419d303241e4b149f92d21d7bc9132a, State: Initialized, Role: FOLLOWER
I20260929 20:32:25.874591 11658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7419d303241e4b149f92d21d7bc9132a" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 37845 } }
I20260929 20:32:25.874888 11658 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:32:25.875023 11658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:32:25.875253 11658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [term 1 FOLLOWER]: Advancing to term 2
I20260929 20:32:25.879949 11658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7419d303241e4b149f92d21d7bc9132a" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 37845 } }
I20260929 20:32:25.880618 11658 leader_election.cc:302] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7419d303241e4b149f92d21d7bc9132a; no voters:
I20260929 20:32:25.881867 11658 leader_election.cc:288] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 20:32:25.881999 11662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [term 2 FOLLOWER]: Leader election won for term 2
I20260929 20:32:25.883246 11662 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [term 2 LEADER]: Becoming Leader. State: Replica: 7419d303241e4b149f92d21d7bc9132a, State: Running, Role: LEADER
I20260929 20:32:25.884136 11662 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7419d303241e4b149f92d21d7bc9132a" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 37845 } }
I20260929 20:32:25.884884 11658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:32:25.887547 11665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7419d303241e4b149f92d21d7bc9132a. Latest consensus state: current_term: 2 leader_uuid: "7419d303241e4b149f92d21d7bc9132a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7419d303241e4b149f92d21d7bc9132a" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 37845 } } }
I20260929 20:32:25.888059 11665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [sys.catalog]: This master's current role is: LEADER
I20260929 20:32:25.887799 11664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7419d303241e4b149f92d21d7bc9132a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7419d303241e4b149f92d21d7bc9132a" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 37845 } } }
I20260929 20:32:25.888530 11664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [sys.catalog]: This master's current role is: LEADER
I20260929 20:32:25.900180 11668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:32:25.909839 11668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:32:25.912256 11668 catalog_manager.cc:1259] Loaded cluster ID: bc694c4e95ae41cebabaea68f9359e61
I20260929 20:32:25.912428 11668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:32:25.917714 11668 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 20:32:25.937512 11668 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 20:32:25.948431 11668 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:32:25.950677 11668 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a: Loaded TSK: 0
I20260929 20:32:25.952657 11668 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 20:32:26.264107 11660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:32:26.264430 11660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:32:26.264593 11660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:32:26.274031 11660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:32:26.274180 11660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:32:26.274375 11660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.129
I20260929 20:32:26.287410 11660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:34391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.129:33535
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.244.129
--webserver_port=45621
--tserver_master_addrs=127.28.244.190:37845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.129
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:32:26.288899 11660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:32:26.290380 11660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:32:26.306854 11687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:26.307547 11686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:26.307781 11689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:32:26.308220 11660 server_base.cc:1034] running on GCE node
I20260929 20:32:26.308998 11660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:32:26.310539 11660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:32:26.311794 11660 hybrid_clock.cc:648] HybridClock initialized: now 1790713946311715 us; error 77 us; skew 500 ppm
I20260929 20:32:26.312258 11660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:32:26 dist-test-slave-p29l krb5kdc[11032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713946, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:32:26.317075 11660 init.cc:377] Logged in from keytab as kudu/127.28.244.129@KRBTEST.COM (short username kudu)
I20260929 20:32:26.319453 11660 webserver.cc:460] Webserver started at http://127.28.244.129:45621/ using document root <none> and password file <none>
I20260929 20:32:26.320567 11660 fs_manager.cc:359] Metadata directory not provided
I20260929 20:32:26.320859 11660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:32:26.329509 11660 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:32:26.333956 11696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:26.336195 11660 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 20:32:26.336439 11660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/wal
uuid: "e1aa120e44bb45e68986e21a8565480e"
format_stamp: "Formatted at 2026-09-29 20:32:22 on dist-test-slave-p29l"
server_key: "897c92fc99c51a8554998a2cb5ad9653"
server_key_iv: "6795ee87173cd6dd9f3872ce37b369af"
server_key_version: "encryptionkey@0"
I20260929 20:32:26.337996 11660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:32:26.362113 11660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:32:26.369624 11660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:32:26.370045 11660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:32:26.371768 11660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:32:26.374475 11660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:32:26.374646 11660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:26.374771 11660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:32:26.374850 11660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:26.414739 11660 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.129:33535
I20260929 20:32:26.414793 11809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.129:33535 every 8 connection(s)
I20260929 20:32:26.416430 11660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
I20260929 20:32:26.421449 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 11660
I20260929 20:32:26.422236 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.130:40679
--local_ip_for_outbound_sockets=127.28.244.130
--tserver_master_addrs=127.28.244.190:37845
--webserver_port=45643
--webserver_interface=127.28.244.130
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:34391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:32:26 dist-test-slave-p29l krb5kdc[11032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713946, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:32:26.452006 11813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:26.420701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.129:56517)
0929 20:32:26.421816 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:26.421874 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:26.423923 (+ 2049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:26.426338 (+ 2415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:26.426347 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:26.426939 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:32:26.427724 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:26.427746 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:26.430752 (+ 3006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:26.430760 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:26.433874 (+ 3114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:26.433908 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:26.435510 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:26.435522 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:26.435993 (+ 471us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:26.436919 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:32:26.436960 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:32:26.440787 (+ 3827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:32:26.445263 (+ 4476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:26.445278 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:26.445302 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:26.446238 (+ 936us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:26.447040 (+ 802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:26.447048 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:26.447055 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:26.447379 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:26.448460 (+ 1081us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:32:26.448473 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:32:26.449024 (+ 551us) client_negotiation.cc:769] Sending connection context
0929 20:32:26.449431 (+ 407us) client_negotiation.cc:240] Negotiation successful
0929 20:32:26.450217 (+ 786us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"thread_start_us":524,"threads_started":1}
I20260929 20:32:26.452152 11812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:26.420363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:56517 (local address 127.28.244.190:37845)
0929 20:32:26.421158 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:26.421189 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:26.422052 (+ 863us) server_negotiation.cc:407] Connection header received
0929 20:32:26.424693 (+ 2641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:26.424713 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:26.425433 (+ 720us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:26.425986 (+ 553us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:26.428066 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:26.430449 (+ 2383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:26.434205 (+ 3756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:26.435157 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:26.435368 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:26.441258 (+ 5890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:26.441269 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:26.441288 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:26.441340 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:26.445019 (+ 3679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:26.446556 (+ 1537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:26.446566 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:26.446573 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:26.446884 (+ 311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:26.447556 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:26.447560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:26.447562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:26.448289 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:26.448469 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:26.449746 (+ 1277us) server_negotiation.cc:299] Negotiation successful
0929 20:32:26.450142 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":342,"threads_started":1}
I20260929 20:32:26.456447 11810 heartbeater.cc:347] Connected to a master server at 127.28.244.190:37845
I20260929 20:32:26.457034 11810 heartbeater.cc:464] Registering TS with master...
I20260929 20:32:26.458683 11810 heartbeater.cc:511] Master 127.28.244.190:37845 requested a full tablet report, sending...
I20260929 20:32:26.461779 11623 ts_manager.cc:194] Registered new tserver with Master: e1aa120e44bb45e68986e21a8565480e (127.28.244.129:33535)
I20260929 20:32:26.463941 11623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.129@KRBTEST.COM'} at 127.28.244.129:56517
W20260929 20:32:26.829972 11814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:32:26.830494 11814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:32:26.830610 11814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:32:26.838531 11814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:32:26.838681 11814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:32:26.838809 11814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.130
I20260929 20:32:26.849180 11814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:34391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.130:40679
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.244.130
--webserver_port=45643
--tserver_master_addrs=127.28.244.190:37845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.130
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:32:26.850549 11814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:32:26.851977 11814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:32:26.867093 11822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:26.867072 11819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:26.867201 11820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:32:26.867910 11814 server_base.cc:1034] running on GCE node
I20260929 20:32:26.868620 11814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:32:26.869732 11814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:32:26.871034 11814 hybrid_clock.cc:648] HybridClock initialized: now 1790713946870980 us; error 38 us; skew 500 ppm
I20260929 20:32:26.871491 11814 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:32:26 dist-test-slave-p29l krb5kdc[11032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713946, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:32:26.875497 11814 init.cc:377] Logged in from keytab as kudu/127.28.244.130@KRBTEST.COM (short username kudu)
I20260929 20:32:26.877475 11814 webserver.cc:460] Webserver started at http://127.28.244.130:45643/ using document root <none> and password file <none>
I20260929 20:32:26.878391 11814 fs_manager.cc:359] Metadata directory not provided
I20260929 20:32:26.878561 11814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:32:26.884912 11814 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 20:32:26.888448 11829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:26.890328 11814 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260929 20:32:26.890543 11814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/wal
uuid: "8312b6a9e28542de82dc41db65e09a35"
format_stamp: "Formatted at 2026-09-29 20:32:23 on dist-test-slave-p29l"
server_key: "76df5c2f91d50add72b4b3d9dba917bf"
server_key_iv: "426b91133a80b2d5b8787c24f1163ab6"
server_key_version: "encryptionkey@0"
I20260929 20:32:26.891669 11814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:32:26.913450 11814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:32:26.919965 11814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:32:26.920324 11814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:32:26.921514 11814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:32:26.923280 11814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:32:26.923384 11814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:26.923578 11814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:32:26.923650 11814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:26.962361 11814 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.130:40679
I20260929 20:32:26.962479 11941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.130:40679 every 8 connection(s)
I20260929 20:32:26.964080 11814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
I20260929 20:32:26.966239 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 11814
I20260929 20:32:26.966871 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.131:40541
--local_ip_for_outbound_sockets=127.28.244.131
--tserver_master_addrs=127.28.244.190:37845
--webserver_port=41021
--webserver_interface=127.28.244.131
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:34391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:32:26 dist-test-slave-p29l krb5kdc[11032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713946, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:32:26.990386 11944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:26.967254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:58073 (local address 127.28.244.190:37845)
0929 20:32:26.967857 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:26.967866 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:26.969035 (+ 1169us) server_negotiation.cc:407] Connection header received
0929 20:32:26.971016 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:26.971026 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:26.971132 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:26.971254 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:26.973173 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:26.973918 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:26.976805 (+ 2887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:26.977247 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:26.977368 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:26.982577 (+ 5209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:26.982592 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:26.982596 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:26.982652 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:26.985983 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:26.987362 (+ 1379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:26.987367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:26.987370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:26.987550 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:26.988223 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:26.988228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:26.988230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:26.988533 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:26.988696 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:26.989859 (+ 1163us) server_negotiation.cc:299] Negotiation successful
0929 20:32:26.990075 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":269,"threads_started":1}
I20260929 20:32:26.992223 11945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:26.967959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.130:58073)
0929 20:32:26.968825 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:26.968882 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:26.970680 (+ 1798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:26.971544 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:26.971555 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:26.972251 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:32:26.972950 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:26.972972 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:26.974124 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:26.974131 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:26.976590 (+ 2459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:26.976613 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:26.977530 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:26.977540 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:26.977929 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:26.978766 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:32:26.978810 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:32:26.982208 (+ 3398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:32:26.986246 (+ 4038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:26.986261 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:26.986288 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:26.987129 (+ 841us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:26.987693 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:26.987702 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:26.987707 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:26.988046 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:26.988682 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:32:26.988694 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:32:26.989323 (+ 629us) client_negotiation.cc:769] Sending connection context
0929 20:32:26.989709 (+ 386us) client_negotiation.cc:240] Negotiation successful
0929 20:32:26.990420 (+ 711us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":365,"threads_started":1}
I20260929 20:32:26.994104 11942 heartbeater.cc:347] Connected to a master server at 127.28.244.190:37845
I20260929 20:32:26.994468 11942 heartbeater.cc:464] Registering TS with master...
I20260929 20:32:26.995326 11942 heartbeater.cc:511] Master 127.28.244.190:37845 requested a full tablet report, sending...
I20260929 20:32:26.997673 11623 ts_manager.cc:194] Registered new tserver with Master: 8312b6a9e28542de82dc41db65e09a35 (127.28.244.130:40679)
I20260929 20:32:26.998868 11623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.130@KRBTEST.COM'} at 127.28.244.130:58073
W20260929 20:32:27.347927 11946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:32:27.348301 11946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:32:27.348476 11946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:32:27.358026 11946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:32:27.358240 11946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:32:27.358443 11946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.131
I20260929 20:32:27.369237 11946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:34391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.131:40541
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.244.131
--webserver_port=41021
--tserver_master_addrs=127.28.244.190:37845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.131
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:32:27.370697 11946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:32:27.372215 11946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:32:27.388375 11952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:27.388384 11953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:27.388428 11955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:32:27.390144 11946 server_base.cc:1034] running on GCE node
I20260929 20:32:27.390945 11946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:32:27.392084 11946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:32:27.393432 11946 hybrid_clock.cc:648] HybridClock initialized: now 1790713947393365 us; error 49 us; skew 500 ppm
I20260929 20:32:27.393847 11946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:32:27 dist-test-slave-p29l krb5kdc[11032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713947, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:32:27.397895 11946 init.cc:377] Logged in from keytab as kudu/127.28.244.131@KRBTEST.COM (short username kudu)
I20260929 20:32:27.399961 11946 webserver.cc:460] Webserver started at http://127.28.244.131:41021/ using document root <none> and password file <none>
I20260929 20:32:27.400825 11946 fs_manager.cc:359] Metadata directory not provided
I20260929 20:32:27.400990 11946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:32:27.407568 11946 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 20:32:27.410956 11962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:27.412891 11946 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 20:32:27.413115 11946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/wal
uuid: "951e78d229d64b709e1b40963e492495"
format_stamp: "Formatted at 2026-09-29 20:32:24 on dist-test-slave-p29l"
server_key: "c450730f0350d3602a4858ebcfe55ff4"
server_key_iv: "f4e6bb5362bdbd100935e6143ad33dca"
server_key_version: "encryptionkey@0"
I20260929 20:32:27.414379 11946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:32:27.443974 11946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:32:27.450567 11946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:32:27.450999 11946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:32:27.452319 11946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:32:27.453986 11946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:32:27.454131 11946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:27.454282 11946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:32:27.454378 11946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:27.468055 11810 heartbeater.cc:503] Master 127.28.244.190:37845 was elected leader, sending a full tablet report...
I20260929 20:32:27.497900 11946 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.131:40541
I20260929 20:32:27.498020 12074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.131:40541 every 8 connection(s)
I20260929 20:32:27.499632 11946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
I20260929 20:32:27.507385 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 11946
Sep 29 20:32:27 dist-test-slave-p29l krb5kdc[11032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713947, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:32:27.527441 12080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:27.508570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56482 (local address 127.28.244.190:37845)
0929 20:32:27.509478 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:27.509484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:27.509506 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:32:27.509567 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:27.509572 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:27.509672 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:27.509787 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:27.512234 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:27.513533 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:27.515672 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:27.516611 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:27.516764 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:27.520388 (+ 3624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:27.520396 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:27.520400 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:27.520465 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:27.524512 (+ 4047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:27.525252 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:27.525260 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:27.525263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:27.525420 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:27.525851 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:27.525857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:27.525862 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:27.526210 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:27.526460 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:27.526808 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 20:32:27.527064 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":771,"thread_start_us":629,"threads_started":1}
I20260929 20:32:27.528110 12077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:27.502850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:58851 (local address 127.28.244.190:37845)
0929 20:32:27.503353 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:27.503357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:27.504744 (+ 1387us) server_negotiation.cc:407] Connection header received
0929 20:32:27.506830 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:27.506836 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:27.506956 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:27.507094 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:27.509938 (+ 2844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:27.511427 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:27.514212 (+ 2785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:27.514879 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:27.515024 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:27.520458 (+ 5434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:27.520464 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:27.520469 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:27.520531 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:27.524293 (+ 3762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:27.525294 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:27.525300 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:27.525303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:27.525442 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:27.525961 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:27.525965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:27.525967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:27.526241 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:27.526406 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:27.527408 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0929 20:32:27.527661 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":274,"threads_started":1}
I20260929 20:32:27.529336 12078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:27.503534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.131:58851)
0929 20:32:27.504540 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:27.504588 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:27.506441 (+ 1853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:27.507390 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:27.507399 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:27.508471 (+ 1072us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:32:27.509515 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:27.509548 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:27.511719 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:27.511724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:27.514014 (+ 2290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:27.514031 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:27.515310 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:27.515318 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:27.515748 (+ 430us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:27.517005 (+ 1257us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:32:27.517033 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:32:27.519871 (+ 2838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:32:27.524515 (+ 4644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:27.524527 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:27.524550 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:27.525105 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:27.525585 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:27.525588 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:27.525591 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:27.525825 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:27.526415 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:32:27.526424 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:32:27.526890 (+ 466us) client_negotiation.cc:769] Sending connection context
0929 20:32:27.527227 (+ 337us) client_negotiation.cc:240] Negotiation successful
0929 20:32:27.527879 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"thread_start_us":528,"threads_started":1}
I20260929 20:32:27.531507 12075 heartbeater.cc:347] Connected to a master server at 127.28.244.190:37845
I20260929 20:32:27.532356 12075 heartbeater.cc:464] Registering TS with master...
I20260929 20:32:27.533533 12075 heartbeater.cc:511] Master 127.28.244.190:37845 requested a full tablet report, sending...
I20260929 20:32:27.536458 11622 ts_manager.cc:194] Registered new tserver with Master: 951e78d229d64b709e1b40963e492495 (127.28.244.131:40541)
I20260929 20:32:27.537554 11622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.131@KRBTEST.COM'} at 127.28.244.131:58851
I20260929 20:32:27.543845 29650 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:32:27.552059 12077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:27.545006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56494 (local address 127.28.244.190:37845)
0929 20:32:27.545197 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:27.545202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:27.545759 (+ 557us) server_negotiation.cc:407] Connection header received
0929 20:32:27.546074 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:27.546079 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:27.546273 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:27.546436 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:32:27.547037 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:27.547797 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:27.549472 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:27.549946 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:27.550064 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:27.550749 (+ 685us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:32:27.551342 (+ 593us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:32:27.551459 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:27.551627 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 20:32:27.551751 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 20:32:27.553267 11622 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:56494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:32:27.562094 11622 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:32:27.602128 12089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:27.578733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39036 (local address 127.28.244.131:40541)
0929 20:32:27.590155 (+ 11422us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:27.590171 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:27.591320 (+ 1149us) server_negotiation.cc:407] Connection header received
0929 20:32:27.591434 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:27.591442 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:27.591768 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:27.591996 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:32:27.593102 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:27.594853 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:27.599114 (+ 4261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:27.600778 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:27.600976 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:27.601157 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:27.601534 (+ 377us) server_negotiation.cc:299] Negotiation successful
0929 20:32:27.601741 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11132,"thread_start_us":10905,"threads_started":1}
I20260929 20:32:27.602329 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:27.578278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:40541 (local address 127.0.0.1:39036)
0929 20:32:27.590821 (+ 12543us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:27.590849 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:27.591166 (+ 317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:27.592329 (+ 1163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:27.592337 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:27.592367 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:32:27.592841 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:27.592862 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:27.595026 (+ 2164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:27.595035 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:27.598872 (+ 3837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:27.598892 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:27.600987 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:27.600994 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:27.601247 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:27.601312 (+ 65us) client_negotiation.cc:769] Sending connection context
0929 20:32:27.601468 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 20:32:27.601701 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12342,"mutex_wait_us":81,"thread_start_us":13070,"threads_started":1}
I20260929 20:32:27.603005 12087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:27.578041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57030 (local address 127.28.244.130:40679)
0929 20:32:27.592037 (+ 13996us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:27.592048 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:27.592080 (+ 32us) server_negotiation.cc:407] Connection header received
0929 20:32:27.592197 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:27.592205 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:27.592441 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:27.592666 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:32:27.593984 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:27.595377 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:27.599093 (+ 3716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:27.600766 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:27.600952 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:27.601157 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:27.602233 (+ 1076us) server_negotiation.cc:299] Negotiation successful
0929 20:32:27.602712 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13807,"thread_start_us":15011,"threads_started":1}
I20260929 20:32:27.603307 12086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:27.577811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:40679 (local address 127.0.0.1:57030)
0929 20:32:27.590716 (+ 12905us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:27.590817 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:27.591058 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:27.592926 (+ 1868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:27.592930 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:27.592961 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:32:27.593320 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:27.593334 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:27.595561 (+ 2227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:27.595567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:27.598889 (+ 3322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:27.598899 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:27.601336 (+ 2437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:27.601341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:27.601513 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:27.601546 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 20:32:27.601637 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 20:32:27.601775 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12587,"mutex_wait_us":42,"thread_start_us":12391,"threads_started":1}
I20260929 20:32:27.604964 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:27.591405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:33535 (local address 127.0.0.1:39820)
0929 20:32:27.594200 (+ 2795us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:27.594221 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:27.594370 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:27.595247 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:27.595251 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:27.595277 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:32:27.596485 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:27.596503 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:27.598162 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:27.598170 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:27.600350 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:27.600364 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:27.602715 (+ 2351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:27.602744 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:27.602936 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:27.602974 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 20:32:27.603171 (+ 197us) client_negotiation.cc:240] Negotiation successful
0929 20:32:27.603330 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2647,"mutex_wait_us":711,"thread_start_us":2480,"threads_started":1}
I20260929 20:32:27.605885 12091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:27.591646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39820 (local address 127.28.244.129:33535)
0929 20:32:27.593855 (+ 2209us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:27.593865 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:27.594343 (+ 478us) server_negotiation.cc:407] Connection header received
0929 20:32:27.594539 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:27.594547 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:27.594768 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:27.594972 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:32:27.596728 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:27.598013 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:27.600536 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:27.602142 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:27.602362 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:27.602558 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:27.605373 (+ 2815us) server_negotiation.cc:299] Negotiation successful
0929 20:32:27.605570 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2016,"thread_start_us":1851,"threads_started":1}
I20260929 20:32:27.607077 11877 tablet_service.cc:1467] Processing CreateTablet for tablet 5110576ee03744dfb0631c5e2ef38dd0 (DEFAULT_TABLE table=test-table [id=3dbdf87ff81e49e5b2cd86b277bdc949]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:32:27.606812 12010 tablet_service.cc:1467] Processing CreateTablet for tablet 5110576ee03744dfb0631c5e2ef38dd0 (DEFAULT_TABLE table=test-table [id=3dbdf87ff81e49e5b2cd86b277bdc949]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:32:27.608444 11877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5110576ee03744dfb0631c5e2ef38dd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:32:27.608683 12010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5110576ee03744dfb0631c5e2ef38dd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:32:27.610180 11744 tablet_service.cc:1467] Processing CreateTablet for tablet 5110576ee03744dfb0631c5e2ef38dd0 (DEFAULT_TABLE table=test-table [id=3dbdf87ff81e49e5b2cd86b277bdc949]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:32:27.612360 11744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5110576ee03744dfb0631c5e2ef38dd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:32:27.622838 12092 tablet_bootstrap.cc:492] T 5110576ee03744dfb0631c5e2ef38dd0 P 8312b6a9e28542de82dc41db65e09a35: Bootstrap starting.
I20260929 20:32:27.626694 12092 tablet_bootstrap.cc:654] T 5110576ee03744dfb0631c5e2ef38dd0 P 8312b6a9e28542de82dc41db65e09a35: Neither blocks nor log segments found. Creating new log.
I20260929 20:32:27.627993 12092 log.cc:824] T 5110576ee03744dfb0631c5e2ef38dd0 P 8312b6a9e28542de82dc41db65e09a35: Log is configured to *not* fsync() on all Append() calls
I20260929 20:32:27.631251 12092 tablet_bootstrap.cc:492] T 5110576ee03744dfb0631c5e2ef38dd0 P 8312b6a9e28542de82dc41db65e09a35: No bootstrap required, opened a new log
I20260929 20:32:27.631561 12092 ts_tablet_manager.cc:1397] T 5110576ee03744dfb0631c5e2ef38dd0 P 8312b6a9e28542de82dc41db65e09a35: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 20:32:27.631996 12093 tablet_bootstrap.cc:492] T 5110576ee03744dfb0631c5e2ef38dd0 P 951e78d229d64b709e1b40963e492495: Bootstrap starting.
I20260929 20:32:27.633132 12095 tablet_bootstrap.cc:492] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e: Bootstrap starting.
I20260929 20:32:27.636904 12093 tablet_bootstrap.cc:654] T 5110576ee03744dfb0631c5e2ef38dd0 P 951e78d229d64b709e1b40963e492495: Neither blocks nor log segments found. Creating new log.
I20260929 20:32:27.636969 12095 tablet_bootstrap.cc:654] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e: Neither blocks nor log segments found. Creating new log.
I20260929 20:32:27.638236 12095 log.cc:824] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e: Log is configured to *not* fsync() on all Append() calls
I20260929 20:32:27.638355 12093 log.cc:824] T 5110576ee03744dfb0631c5e2ef38dd0 P 951e78d229d64b709e1b40963e492495: Log is configured to *not* fsync() on all Append() calls
I20260929 20:32:27.641875 12093 tablet_bootstrap.cc:492] T 5110576ee03744dfb0631c5e2ef38dd0 P 951e78d229d64b709e1b40963e492495: No bootstrap required, opened a new log
I20260929 20:32:27.642064 12095 tablet_bootstrap.cc:492] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e: No bootstrap required, opened a new log
I20260929 20:32:27.642287 12093 ts_tablet_manager.cc:1397] T 5110576ee03744dfb0631c5e2ef38dd0 P 951e78d229d64b709e1b40963e492495: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260929 20:32:27.642400 12095 ts_tablet_manager.cc:1397] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260929 20:32:27.643908 12092 raft_consensus.cc:348] T 5110576ee03744dfb0631c5e2ef38dd0 P 8312b6a9e28542de82dc41db65e09a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8312b6a9e28542de82dc41db65e09a35" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40679 } } peers { permanent_uuid: "e1aa120e44bb45e68986e21a8565480e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 33535 } } peers { permanent_uuid: "951e78d229d64b709e1b40963e492495" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 40541 } }
I20260929 20:32:27.644448 12092 raft_consensus.cc:374] T 5110576ee03744dfb0631c5e2ef38dd0 P 8312b6a9e28542de82dc41db65e09a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:32:27.644551 12092 raft_consensus.cc:729] T 5110576ee03744dfb0631c5e2ef38dd0 P 8312b6a9e28542de82dc41db65e09a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8312b6a9e28542de82dc41db65e09a35, State: Initialized, Role: FOLLOWER
I20260929 20:32:27.645298 12092 consensus_queue.cc:260] T 5110576ee03744dfb0631c5e2ef38dd0 P 8312b6a9e28542de82dc41db65e09a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8312b6a9e28542de82dc41db65e09a35" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40679 } } peers { permanent_uuid: "e1aa120e44bb45e68986e21a8565480e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 33535 } } peers { permanent_uuid: "951e78d229d64b709e1b40963e492495" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 40541 } }
I20260929 20:32:27.648693 11942 heartbeater.cc:503] Master 127.28.244.190:37845 was elected leader, sending a full tablet report...
I20260929 20:32:27.649569 12092 ts_tablet_manager.cc:1428] T 5110576ee03744dfb0631c5e2ef38dd0 P 8312b6a9e28542de82dc41db65e09a35: Time spent starting tablet: real 0.018s user 0.011s sys 0.007s
I20260929 20:32:27.653544 12093 raft_consensus.cc:348] T 5110576ee03744dfb0631c5e2ef38dd0 P 951e78d229d64b709e1b40963e492495 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8312b6a9e28542de82dc41db65e09a35" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40679 } } peers { permanent_uuid: "e1aa120e44bb45e68986e21a8565480e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 33535 } } peers { permanent_uuid: "951e78d229d64b709e1b40963e492495" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 40541 } }
I20260929 20:32:27.654008 12093 raft_consensus.cc:374] T 5110576ee03744dfb0631c5e2ef38dd0 P 951e78d229d64b709e1b40963e492495 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:32:27.654140 12093 raft_consensus.cc:729] T 5110576ee03744dfb0631c5e2ef38dd0 P 951e78d229d64b709e1b40963e492495 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951e78d229d64b709e1b40963e492495, State: Initialized, Role: FOLLOWER
I20260929 20:32:27.654383 12095 raft_consensus.cc:348] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8312b6a9e28542de82dc41db65e09a35" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40679 } } peers { permanent_uuid: "e1aa120e44bb45e68986e21a8565480e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 33535 } } peers { permanent_uuid: "951e78d229d64b709e1b40963e492495" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 40541 } }
I20260929 20:32:27.654945 12095 raft_consensus.cc:374] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:32:27.655089 12095 raft_consensus.cc:729] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1aa120e44bb45e68986e21a8565480e, State: Initialized, Role: FOLLOWER
I20260929 20:32:27.654876 12093 consensus_queue.cc:260] T 5110576ee03744dfb0631c5e2ef38dd0 P 951e78d229d64b709e1b40963e492495 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8312b6a9e28542de82dc41db65e09a35" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40679 } } peers { permanent_uuid: "e1aa120e44bb45e68986e21a8565480e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 33535 } } peers { permanent_uuid: "951e78d229d64b709e1b40963e492495" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 40541 } }
I20260929 20:32:27.655767 12095 consensus_queue.cc:260] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8312b6a9e28542de82dc41db65e09a35" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40679 } } peers { permanent_uuid: "e1aa120e44bb45e68986e21a8565480e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 33535 } } peers { permanent_uuid: "951e78d229d64b709e1b40963e492495" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 40541 } }
I20260929 20:32:27.656913 12075 heartbeater.cc:503] Master 127.28.244.190:37845 was elected leader, sending a full tablet report...
I20260929 20:32:27.657667 12093 ts_tablet_manager.cc:1428] T 5110576ee03744dfb0631c5e2ef38dd0 P 951e78d229d64b709e1b40963e492495: Time spent starting tablet: real 0.015s user 0.012s sys 0.005s
I20260929 20:32:27.660482 12095 ts_tablet_manager.cc:1428] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e: Time spent starting tablet: real 0.018s user 0.015s sys 0.005s
W20260929 20:32:27.669852 11811 tablet.cc:2369] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:32:27.716351 11943 tablet.cc:2369] T 5110576ee03744dfb0631c5e2ef38dd0 P 8312b6a9e28542de82dc41db65e09a35: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:32:27.740020 12099 raft_consensus.cc:482] T 5110576ee03744dfb0631c5e2ef38dd0 P 951e78d229d64b709e1b40963e492495 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:32:27.740388 12099 raft_consensus.cc:504] T 5110576ee03744dfb0631c5e2ef38dd0 P 951e78d229d64b709e1b40963e492495 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8312b6a9e28542de82dc41db65e09a35" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40679 } } peers { permanent_uuid: "e1aa120e44bb45e68986e21a8565480e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 33535 } } peers { permanent_uuid: "951e78d229d64b709e1b40963e492495" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 40541 } }
I20260929 20:32:27.742189 12099 leader_election.cc:288] T 5110576ee03744dfb0631c5e2ef38dd0 P 951e78d229d64b709e1b40963e492495 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8312b6a9e28542de82dc41db65e09a35 (127.28.244.130:40679), e1aa120e44bb45e68986e21a8565480e (127.28.244.129:33535)
I20260929 20:32:27.750111 12087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:27.742835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:44531 (local address 127.28.244.130:40679)
0929 20:32:27.743088 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:27.743095 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:27.743118 (+ 23us) server_negotiation.cc:407] Connection header received
0929 20:32:27.743334 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:27.743338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:27.743398 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:27.743504 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:32:27.743994 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:27.745097 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:27.747764 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:27.749062 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:27.749215 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:27.749280 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:27.749727 (+ 447us) server_negotiation.cc:299] Negotiation successful
0929 20:32:27.749884 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 20:32:27.750169 12078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:27.742690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:40679 (local address 127.28.244.131:44531)
0929 20:32:27.742938 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:27.742964 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:27.743165 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:27.743525 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:27.743529 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:27.743557 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:32:27.743787 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:27.743837 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:27.745269 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:27.745278 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:27.747485 (+ 2207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:27.747502 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:27.749252 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:27.749259 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:27.749481 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:27.749524 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 20:32:27.749646 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 20:32:27.749831 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":1}
I20260929 20:32:27.751619 12091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:27.743436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:48209 (local address 127.28.244.129:33535)
0929 20:32:27.743895 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:27.743903 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:27.744165 (+ 262us) server_negotiation.cc:407] Connection header received
0929 20:32:27.744394 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:27.744399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:27.744465 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:27.744556 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:32:27.745109 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:27.746420 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:27.749129 (+ 2709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:27.750577 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:27.750786 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:27.750867 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:27.751203 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 20:32:27.751344 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277}
I20260929 20:32:27.751600 11897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5110576ee03744dfb0631c5e2ef38dd0" candidate_uuid: "951e78d229d64b709e1b40963e492495" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8312b6a9e28542de82dc41db65e09a35" is_pre_election: true
I20260929 20:32:27.752005 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:27.743129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:33535 (local address 127.28.244.131:48209)
0929 20:32:27.744028 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:27.744048 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:27.744209 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:27.744596 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:27.744601 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:27.744628 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:32:27.744940 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:27.744952 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:27.746627 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:27.746631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:27.748906 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:27.748927 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:27.750799 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:27.750805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:27.750996 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:27.751048 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 20:32:27.751172 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 20:32:27.751334 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"mutex_wait_us":1,"thread_start_us":582,"threads_started":1}
I20260929 20:32:27.752341 11897 raft_consensus.cc:2454] T 5110576ee03744dfb0631c5e2ef38dd0 P 8312b6a9e28542de82dc41db65e09a35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 951e78d229d64b709e1b40963e492495 in term 0.
W20260929 20:32:27.752538 12076 tablet.cc:2369] T 5110576ee03744dfb0631c5e2ef38dd0 P 951e78d229d64b709e1b40963e492495: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:32:27.753212 11964 leader_election.cc:302] T 5110576ee03744dfb0631c5e2ef38dd0 P 951e78d229d64b709e1b40963e492495 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8312b6a9e28542de82dc41db65e09a35, 951e78d229d64b709e1b40963e492495; no voters:
I20260929 20:32:27.753230 11764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5110576ee03744dfb0631c5e2ef38dd0" candidate_uuid: "951e78d229d64b709e1b40963e492495" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1aa120e44bb45e68986e21a8565480e" is_pre_election: true
I20260929 20:32:27.753710 12099 raft_consensus.cc:2781] T 5110576ee03744dfb0631c5e2ef38dd0 P 951e78d229d64b709e1b40963e492495 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:32:27.753851 12099 raft_consensus.cc:482] T 5110576ee03744dfb0631c5e2ef38dd0 P 951e78d229d64b709e1b40963e492495 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:32:27.753764 11764 raft_consensus.cc:2454] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 951e78d229d64b709e1b40963e492495 in term 0.
I20260929 20:32:27.754004 12099 raft_consensus.cc:3037] T 5110576ee03744dfb0631c5e2ef38dd0 P 951e78d229d64b709e1b40963e492495 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:32:27.756688 12099 raft_consensus.cc:504] T 5110576ee03744dfb0631c5e2ef38dd0 P 951e78d229d64b709e1b40963e492495 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8312b6a9e28542de82dc41db65e09a35" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40679 } } peers { permanent_uuid: "e1aa120e44bb45e68986e21a8565480e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 33535 } } peers { permanent_uuid: "951e78d229d64b709e1b40963e492495" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 40541 } }
I20260929 20:32:27.757438 12099 leader_election.cc:288] T 5110576ee03744dfb0631c5e2ef38dd0 P 951e78d229d64b709e1b40963e492495 [CANDIDATE]: Term 1 election: Requested vote from peers 8312b6a9e28542de82dc41db65e09a35 (127.28.244.130:40679), e1aa120e44bb45e68986e21a8565480e (127.28.244.129:33535)
I20260929 20:32:27.758117 11897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5110576ee03744dfb0631c5e2ef38dd0" candidate_uuid: "951e78d229d64b709e1b40963e492495" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8312b6a9e28542de82dc41db65e09a35"
I20260929 20:32:27.758189 11764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5110576ee03744dfb0631c5e2ef38dd0" candidate_uuid: "951e78d229d64b709e1b40963e492495" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1aa120e44bb45e68986e21a8565480e"
I20260929 20:32:27.758373 11897 raft_consensus.cc:3037] T 5110576ee03744dfb0631c5e2ef38dd0 P 8312b6a9e28542de82dc41db65e09a35 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:32:27.758502 11764 raft_consensus.cc:3037] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:32:27.761025 11897 raft_consensus.cc:2454] T 5110576ee03744dfb0631c5e2ef38dd0 P 8312b6a9e28542de82dc41db65e09a35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 951e78d229d64b709e1b40963e492495 in term 1.
I20260929 20:32:27.761171 11764 raft_consensus.cc:2454] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 951e78d229d64b709e1b40963e492495 in term 1.
I20260929 20:32:27.761621 11964 leader_election.cc:302] T 5110576ee03744dfb0631c5e2ef38dd0 P 951e78d229d64b709e1b40963e492495 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8312b6a9e28542de82dc41db65e09a35, 951e78d229d64b709e1b40963e492495; no voters:
I20260929 20:32:27.762024 12099 raft_consensus.cc:2781] T 5110576ee03744dfb0631c5e2ef38dd0 P 951e78d229d64b709e1b40963e492495 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:32:27.762511 12099 raft_consensus.cc:686] T 5110576ee03744dfb0631c5e2ef38dd0 P 951e78d229d64b709e1b40963e492495 [term 1 LEADER]: Becoming Leader. State: Replica: 951e78d229d64b709e1b40963e492495, State: Running, Role: LEADER
I20260929 20:32:27.763032 12099 consensus_queue.cc:237] T 5110576ee03744dfb0631c5e2ef38dd0 P 951e78d229d64b709e1b40963e492495 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8312b6a9e28542de82dc41db65e09a35" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40679 } } peers { permanent_uuid: "e1aa120e44bb45e68986e21a8565480e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 33535 } } peers { permanent_uuid: "951e78d229d64b709e1b40963e492495" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 40541 } }
I20260929 20:32:27.768733 11622 catalog_manager.cc:5495] T 5110576ee03744dfb0631c5e2ef38dd0 P 951e78d229d64b709e1b40963e492495 reported cstate change: term changed from 0 to 1, leader changed from <none> to 951e78d229d64b709e1b40963e492495 (127.28.244.131). New cstate: current_term: 1 leader_uuid: "951e78d229d64b709e1b40963e492495" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8312b6a9e28542de82dc41db65e09a35" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1aa120e44bb45e68986e21a8565480e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 33535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "951e78d229d64b709e1b40963e492495" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 40541 } health_report { overall_health: HEALTHY } } }
I20260929 20:32:27.804325 12089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:27.797187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39044 (local address 127.28.244.131:40541)
0929 20:32:27.797430 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:27.797435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:27.797453 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:32:27.797693 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:27.797698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:27.797816 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:27.797975 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:32:27.798458 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:27.799251 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:27.801436 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:27.801987 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:27.802160 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:27.802756 (+ 596us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:32:27.803506 (+ 750us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:32:27.803664 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:27.803918 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 20:32:27.804019 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 20:32:27.815070 11764 raft_consensus.cc:1261] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [term 1 FOLLOWER]: Refusing update from remote peer 951e78d229d64b709e1b40963e492495: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:32:27.815271 11897 raft_consensus.cc:1261] T 5110576ee03744dfb0631c5e2ef38dd0 P 8312b6a9e28542de82dc41db65e09a35 [term 1 FOLLOWER]: Refusing update from remote peer 951e78d229d64b709e1b40963e492495: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:32:27.816490 12102 consensus_queue.cc:1035] T 5110576ee03744dfb0631c5e2ef38dd0 P 951e78d229d64b709e1b40963e492495 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8312b6a9e28542de82dc41db65e09a35" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:32:27.816949 12099 consensus_queue.cc:1035] T 5110576ee03744dfb0631c5e2ef38dd0 P 951e78d229d64b709e1b40963e492495 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1aa120e44bb45e68986e21a8565480e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 33535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:32:27.831452 12105 mvcc.cc:205] Tried to move back new op lower bound from 7334764330238152704 to 7334764330041942016. Current Snapshot: MvccSnapshot[applied={T|T < 7334764330238152704}]
I20260929 20:32:27.837206 12106 mvcc.cc:205] Tried to move back new op lower bound from 7334764330238152704 to 7334764330041942016. Current Snapshot: MvccSnapshot[applied={T|T < 7334764330238152704}]
I20260929 20:32:27.847164 11623 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:56494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:32:27.847510 11623 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:56494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:32:27.852375 11623 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a: Sending DeleteTablet for 3 replicas of tablet 5110576ee03744dfb0631c5e2ef38dd0
I20260929 20:32:27.853988 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 11660
I20260929 20:32:27.853960 11877 tablet_service.cc:1514] Processing DeleteTablet for tablet 5110576ee03744dfb0631c5e2ef38dd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:32:27 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:57030
I20260929 20:32:27.853955 12010 tablet_service.cc:1514] Processing DeleteTablet for tablet 5110576ee03744dfb0631c5e2ef38dd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:32:27 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:39036
I20260929 20:32:27.854205 11744 tablet_service.cc:1514] Processing DeleteTablet for tablet 5110576ee03744dfb0631c5e2ef38dd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:32:27 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:39820
I20260929 20:32:27.876665 12115 tablet_replica.cc:330] T 5110576ee03744dfb0631c5e2ef38dd0 P 951e78d229d64b709e1b40963e492495: stopping tablet replica
I20260929 20:32:27.877386 12115 raft_consensus.cc:2229] T 5110576ee03744dfb0631c5e2ef38dd0 P 951e78d229d64b709e1b40963e492495 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:32:27.878933 12116 tablet_replica.cc:330] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e: stopping tablet replica
I20260929 20:32:27.879492 12116 raft_consensus.cc:2229] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:32:27.880301 12116 raft_consensus.cc:2258] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:32:27.883077 12116 ts_tablet_manager.cc:1905] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:32:27.884595 12091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:27.855986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39822 (local address 127.28.244.129:33535)
0929 20:32:27.856225 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:27.856234 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:27.856260 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:32:27.856338 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:27.856346 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:27.856452 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:27.856614 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:27.859373 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:27.861067 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:27.863993 (+ 2926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:27.864993 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:27.865178 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:27.870691 (+ 5513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:27.870723 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:27.870758 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:27.870832 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:27.878007 (+ 7175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:27.879513 (+ 1506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:27.879529 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:27.879535 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:27.879854 (+ 319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:27.880499 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:27.880508 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:27.880514 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:27.881191 (+ 677us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:27.881662 (+ 471us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:27.883574 (+ 1912us) server_negotiation.cc:299] Negotiation successful
0929 20:32:27.883972 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 20:32:27.885668 11804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:32:27.886322 12115 raft_consensus.cc:2258] T 5110576ee03744dfb0631c5e2ef38dd0 P 951e78d229d64b709e1b40963e492495 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:32:27.887416 12114 tablet_replica.cc:330] T 5110576ee03744dfb0631c5e2ef38dd0 P 8312b6a9e28542de82dc41db65e09a35: stopping tablet replica
I20260929 20:32:27.888075 12114 raft_consensus.cc:2229] T 5110576ee03744dfb0631c5e2ef38dd0 P 8312b6a9e28542de82dc41db65e09a35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:32:27.888710 12114 raft_consensus.cc:2258] T 5110576ee03744dfb0631c5e2ef38dd0 P 8312b6a9e28542de82dc41db65e09a35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:32:27.890173 12115 ts_tablet_manager.cc:1905] T 5110576ee03744dfb0631c5e2ef38dd0 P 951e78d229d64b709e1b40963e492495: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:32:27.890807 12114 ts_tablet_manager.cc:1905] T 5110576ee03744dfb0631c5e2ef38dd0 P 8312b6a9e28542de82dc41db65e09a35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:32:27.900008 12114 ts_tablet_manager.cc:1918] T 5110576ee03744dfb0631c5e2ef38dd0 P 8312b6a9e28542de82dc41db65e09a35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:32:27.900261 12114 log.cc:1196] T 5110576ee03744dfb0631c5e2ef38dd0 P 8312b6a9e28542de82dc41db65e09a35: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/wal/wals/5110576ee03744dfb0631c5e2ef38dd0
I20260929 20:32:27.900585 12115 ts_tablet_manager.cc:1918] T 5110576ee03744dfb0631c5e2ef38dd0 P 951e78d229d64b709e1b40963e492495: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:32:27.900964 12114 ts_tablet_manager.cc:1939] T 5110576ee03744dfb0631c5e2ef38dd0 P 8312b6a9e28542de82dc41db65e09a35: Deleting consensus metadata
I20260929 20:32:27.900985 12115 log.cc:1196] T 5110576ee03744dfb0631c5e2ef38dd0 P 951e78d229d64b709e1b40963e492495: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/wal/wals/5110576ee03744dfb0631c5e2ef38dd0
I20260929 20:32:27.901893 12115 ts_tablet_manager.cc:1939] T 5110576ee03744dfb0631c5e2ef38dd0 P 951e78d229d64b709e1b40963e492495: Deleting consensus metadata
I20260929 20:32:27.903576 11608 catalog_manager.cc:4841] TS 8312b6a9e28542de82dc41db65e09a35 (127.28.244.130:40679): tablet 5110576ee03744dfb0631c5e2ef38dd0 (table test-table [id=3dbdf87ff81e49e5b2cd86b277bdc949]) successfully deleted
I20260929 20:32:27.904068 11607 catalog_manager.cc:4841] TS 951e78d229d64b709e1b40963e492495 (127.28.244.131:40541): tablet 5110576ee03744dfb0631c5e2ef38dd0 (table test-table [id=3dbdf87ff81e49e5b2cd86b277bdc949]) successfully deleted
I20260929 20:32:27.991050 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 11660
W20260929 20:32:28.015473 11608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.244.129:33535 (error 108)
I20260929 20:32:28.016203 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 11814
I20260929 20:32:28.018344 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:28.017767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:33535 (local address 127.0.0.1:39830)
0929 20:32:28.018028 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:28.018164 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:33535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
W20260929 20:32:28.018646 11608 catalog_manager.cc:4584] TS e1aa120e44bb45e68986e21a8565480e (127.28.244.129:33535): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5110576ee03744dfb0631c5e2ef38dd0: Network error: Client connection negotiation failed: client connection to 127.28.244.129:33535: connect: Connection refused (error 111)
I20260929 20:32:28.033219 12087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:28.017067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57044 (local address 127.28.244.130:40679)
0929 20:32:28.017291 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:28.017298 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:28.017325 (+ 27us) server_negotiation.cc:407] Connection header received
0929 20:32:28.017530 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:28.017538 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:28.017645 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:28.017780 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:28.019848 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:28.020746 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:28.022483 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:28.022954 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:28.023110 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:28.026066 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:28.026083 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:28.026100 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:28.026142 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:28.029882 (+ 3740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:28.030800 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:28.030808 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:28.030811 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:28.030953 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:28.031408 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:28.031413 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:28.031418 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:28.031911 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:28.032227 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:28.032604 (+ 377us) server_negotiation.cc:299] Negotiation successful
0929 20:32:28.032860 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 20:32:28.033890 11937 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:32:28.072751 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:28.072123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:33535 (local address 127.0.0.1:39840)
0929 20:32:28.072391 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:28.072520 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:33535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260929 20:32:28.074025 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:28.073581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:33535 (local address 127.0.0.1:39842)
0929 20:32:28.073756 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:28.073871 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:33535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 20:32:28.132170 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 11814
I20260929 20:32:28.134859 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:28.134219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:33535 (local address 127.0.0.1:39856)
0929 20:32:28.134497 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:28.134625 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:33535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260929 20:32:28.136669 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:28.136198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:33535 (local address 127.0.0.1:39862)
0929 20:32:28.136398 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:28.136503 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:33535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 20:32:28.157462 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 11946
I20260929 20:32:28.176549 12089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:28.158259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39050 (local address 127.28.244.131:40541)
0929 20:32:28.158471 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:28.158476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:28.159085 (+ 609us) server_negotiation.cc:407] Connection header received
0929 20:32:28.159249 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:28.159253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:28.159360 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:28.159461 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:28.161516 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:28.162550 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:28.164675 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:28.165131 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:28.165252 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:28.168934 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:28.168948 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:28.168965 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:28.169011 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:28.172942 (+ 3931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:28.173989 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:28.173997 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:28.174000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:28.174181 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:28.174633 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:28.174637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:28.174639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:28.175051 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:28.175313 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:28.175767 (+ 454us) server_negotiation.cc:299] Negotiation successful
0929 20:32:28.176068 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 20:32:28.177354 12070 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:32:28.217914 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:28.217259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:33535 (local address 127.0.0.1:39878)
0929 20:32:28.217521 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:28.217683 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:33535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260929 20:32:28.219408 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:28.218950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:33535 (local address 127.0.0.1:39882)
0929 20:32:28.219196 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:28.219282 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:33535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260929 20:32:28.307291 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 11946
I20260929 20:32:28.329310 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 11590
I20260929 20:32:28.330890 11653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:32:28.455184 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:28.454526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:33535 (local address 127.0.0.1:39884)
0929 20:32:28.454740 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:28.454882 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:33535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 20:32:28.455569 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 11590
I20260929 20:32:28.480451 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:37845
--webserver_interface=127.28.244.190
--webserver_port=40263
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:34391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:37845
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:32:28.863970 12122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:32:28.864326 12122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:32:28.864423 12122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:32:28.872879 12122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:32:28.873044 12122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:32:28.873098 12122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:32:28.873149 12122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:32:28.873189 12122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:32:28.885632 12122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:34391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:37845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:37845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=40263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:32:28.887372 12122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:32:28.888950 12122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:32:28.903264 12128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:28.903301 12131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:28.903064 12129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:32:28.909686 12122 server_base.cc:1034] running on GCE node
I20260929 20:32:28.910748 12122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:32:28.912599 12122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:32:28.914000 12122 hybrid_clock.cc:648] HybridClock initialized: now 1790713948913927 us; error 87 us; skew 500 ppm
I20260929 20:32:28.914494 12122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:32:28 dist-test-slave-p29l krb5kdc[11032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713948, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:32:28.918521 12122 init.cc:377] Logged in from keytab as kudu/127.28.244.190@KRBTEST.COM (short username kudu)
I20260929 20:32:28.920563 12122 webserver.cc:460] Webserver started at http://127.28.244.190:40263/ using document root <none> and password file <none>
I20260929 20:32:28.921412 12122 fs_manager.cc:359] Metadata directory not provided
I20260929 20:32:28.921576 12122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:32:28.928365 12122 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 20:32:28.933929 12138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:28.935701 12122 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 20:32:28.935993 12122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "7419d303241e4b149f92d21d7bc9132a"
format_stamp: "Formatted at 2026-09-29 20:32:22 on dist-test-slave-p29l"
server_key: "4b0c1ac5b33c319c2789b0ab2aad0838"
server_key_iv: "9561148bfb031b03b0d3b191f5ee30e1"
server_key_version: "encryptionkey@0"
I20260929 20:32:28.937158 12122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:32:28.953058 12122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:32:28.958158 12122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:32:28.958419 12122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:32:28.979907 12122 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:37845
I20260929 20:32:28.980002 12190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:37845 every 8 connection(s)
I20260929 20:32:28.981588 12122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:32:28.988547 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 12122
I20260929 20:32:28.989516 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.129:33535
--local_ip_for_outbound_sockets=127.28.244.129
--tserver_master_addrs=127.28.244.190:37845
--webserver_port=45621
--webserver_interface=127.28.244.129
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:34391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:32:28.990588 12191 sys_catalog.cc:263] Verifying existing consensus state
I20260929 20:32:28.994549 12191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a: Bootstrap starting.
I20260929 20:32:29.025146 12191 log.cc:824] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a: Log is configured to *not* fsync() on all Append() calls
I20260929 20:32:29.044284 12191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:32:29.044914 12191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a: Bootstrap complete.
I20260929 20:32:29.054562 12191 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7419d303241e4b149f92d21d7bc9132a" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 37845 } }
I20260929 20:32:29.055537 12191 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7419d303241e4b149f92d21d7bc9132a, State: Initialized, Role: FOLLOWER
I20260929 20:32:29.056363 12191 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7419d303241e4b149f92d21d7bc9132a" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 37845 } }
I20260929 20:32:29.056625 12191 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:32:29.056748 12191 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:32:29.056936 12191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [term 2 FOLLOWER]: Advancing to term 3
I20260929 20:32:29.061400 12191 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7419d303241e4b149f92d21d7bc9132a" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 37845 } }
I20260929 20:32:29.062043 12191 leader_election.cc:302] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7419d303241e4b149f92d21d7bc9132a; no voters:
I20260929 20:32:29.062803 12191 leader_election.cc:288] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 20:32:29.062971 12195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [term 3 FOLLOWER]: Leader election won for term 3
I20260929 20:32:29.065084 12195 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [term 3 LEADER]: Becoming Leader. State: Replica: 7419d303241e4b149f92d21d7bc9132a, State: Running, Role: LEADER
I20260929 20:32:29.066023 12191 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:32:29.065939 12195 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7419d303241e4b149f92d21d7bc9132a" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 37845 } }
I20260929 20:32:29.068813 12197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7419d303241e4b149f92d21d7bc9132a. Latest consensus state: current_term: 3 leader_uuid: "7419d303241e4b149f92d21d7bc9132a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7419d303241e4b149f92d21d7bc9132a" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 37845 } } }
I20260929 20:32:29.069458 12196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "7419d303241e4b149f92d21d7bc9132a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7419d303241e4b149f92d21d7bc9132a" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 37845 } } }
I20260929 20:32:29.069921 12197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [sys.catalog]: This master's current role is: LEADER
I20260929 20:32:29.069902 12196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7419d303241e4b149f92d21d7bc9132a [sys.catalog]: This master's current role is: LEADER
I20260929 20:32:29.072412 12203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:32:29.084812 12203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:32:29.087306 12203 catalog_manager.cc:1259] Loaded cluster ID: bc694c4e95ae41cebabaea68f9359e61
I20260929 20:32:29.087575 12203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 20:32:29.092736 12203 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f87a7d3d9ad google::LogMessage::Fail() at ??:0
@ 0x7f87a7d4088a google::LogMessage::SendToLog() at ??:0
@ 0x7f87a7d3d4de google::LogMessage::Flush() at ??:0
@ 0x7f87a7d3ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f87bd774512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f87bd771976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f87bd7f9610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f87bd7f9439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f87aa989345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f87aa98c91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f87aa98c799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f87aa967eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f87adc416db start_thread at ??:0
@ 0x7f87a6bd761f clone at ??:0
W20260929 20:32:29.404140 12193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:32:29.404520 12193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:32:29.404703 12193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:32:29.412817 12193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:32:29.412993 12193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:32:29.413190 12193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.129
I20260929 20:32:29.423708 12193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:34391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.129:33535
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.244.129
--webserver_port=45621
--tserver_master_addrs=127.28.244.190:37845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.129
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:32:29.425230 12193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:32:29.426666 12193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:32:29.442250 12233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:29.442338 12236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:29.442340 12234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:32:29.443202 12193 server_base.cc:1034] running on GCE node
I20260929 20:32:29.444073 12193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:32:29.445274 12193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:32:29.446518 12193 hybrid_clock.cc:648] HybridClock initialized: now 1790713949446434 us; error 69 us; skew 500 ppm
I20260929 20:32:29.446913 12193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:32:29 dist-test-slave-p29l krb5kdc[11032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713949, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:32:29.450459 12193 init.cc:377] Logged in from keytab as kudu/127.28.244.129@KRBTEST.COM (short username kudu)
I20260929 20:32:29.452337 12193 webserver.cc:460] Webserver started at http://127.28.244.129:45621/ using document root <none> and password file <none>
I20260929 20:32:29.453132 12193 fs_manager.cc:359] Metadata directory not provided
I20260929 20:32:29.453296 12193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:32:29.459313 12193 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 20:32:29.463307 12243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:29.465165 12193 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260929 20:32:29.465382 12193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/wal
uuid: "e1aa120e44bb45e68986e21a8565480e"
format_stamp: "Formatted at 2026-09-29 20:32:22 on dist-test-slave-p29l"
server_key: "897c92fc99c51a8554998a2cb5ad9653"
server_key_iv: "6795ee87173cd6dd9f3872ce37b369af"
server_key_version: "encryptionkey@0"
I20260929 20:32:29.466480 12193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:32:29.502024 12193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:32:29.507022 12193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:32:29.507364 12193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:32:29.508620 12193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:32:29.510810 12250 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260929 20:32:29.518033 12193 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 20:32:29.518230 12193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.001s sys 0.000s
I20260929 20:32:29.518386 12193 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 20:32:29.522071 12193 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 20:32:29.522204 12193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.002s sys 0.000s
I20260929 20:32:29.522608 12250 tablet_bootstrap.cc:492] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e: Bootstrap starting.
I20260929 20:32:29.532949 12250 log.cc:824] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e: Log is configured to *not* fsync() on all Append() calls
I20260929 20:32:29.553221 12250 tablet_bootstrap.cc:492] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:32:29.554008 12250 tablet_bootstrap.cc:492] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e: Bootstrap complete.
I20260929 20:32:29.554549 12250 ts_tablet_manager.cc:1397] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e: Time spent bootstrapping tablet: real 0.032s user 0.021s sys 0.007s
I20260929 20:32:29.566634 12250 raft_consensus.cc:348] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8312b6a9e28542de82dc41db65e09a35" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40679 } } peers { permanent_uuid: "e1aa120e44bb45e68986e21a8565480e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 33535 } } peers { permanent_uuid: "951e78d229d64b709e1b40963e492495" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 40541 } }
I20260929 20:32:29.568446 12250 raft_consensus.cc:729] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1aa120e44bb45e68986e21a8565480e, State: Initialized, Role: FOLLOWER
I20260929 20:32:29.569042 12193 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.129:33535
I20260929 20:32:29.569097 12358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.129:33535 every 8 connection(s)
I20260929 20:32:29.569422 12250 consensus_queue.cc:260] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8312b6a9e28542de82dc41db65e09a35" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40679 } } peers { permanent_uuid: "e1aa120e44bb45e68986e21a8565480e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 33535 } } peers { permanent_uuid: "951e78d229d64b709e1b40963e492495" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 40541 } }
I20260929 20:32:29.570550 12193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
I20260929 20:32:29.571640 12250 ts_tablet_manager.cc:1428] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e: Time spent starting tablet: real 0.017s user 0.013s sys 0.004s
I20260929 20:32:29.573259 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 12193
I20260929 20:32:29.573911 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.130:40679
--local_ip_for_outbound_sockets=127.28.244.130
--tserver_master_addrs=127.28.244.190:37845
--webserver_port=45643
--webserver_interface=127.28.244.130
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:34391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:32:29.576136 12362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:29.573731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.129:39015)
0929 20:32:29.574759 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:29.575100 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":715,"thread_start_us":505,"threads_started":1}
W20260929 20:32:29.576889 12360 heartbeater.cc:644] Failed to heartbeat to 127.28.244.190:37845 (0 consecutive failures): Network error: Failed to ping master at 127.28.244.190:37845: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
I20260929 20:32:29.577991 12362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:29.577570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.129:40529)
0929 20:32:29.577784 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:29.577865 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 20:32:29.578933 12362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:29.578560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.129:58017)
0929 20:32:29.578717 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:29.578808 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 20:32:29.579151 12360 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 20:32:29.929705 12363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:32:29.930032 12363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:32:29.930204 12363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:32:29.938187 12363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:32:29.938336 12363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:32:29.938539 12363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.130
I20260929 20:32:29.949502 12363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:34391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.130:40679
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.244.130
--webserver_port=45643
--tserver_master_addrs=127.28.244.190:37845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.130
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:32:29.950944 12363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:32:29.952378 12363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:32:29.967724 12368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:29.967778 12371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:29.967694 12369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:32:29.968443 12363 server_base.cc:1034] running on GCE node
I20260929 20:32:29.969152 12363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:32:29.970280 12363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:32:29.971566 12363 hybrid_clock.cc:648] HybridClock initialized: now 1790713949971509 us; error 40 us; skew 500 ppm
I20260929 20:32:29.971942 12363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:32:29 dist-test-slave-p29l krb5kdc[11032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713949, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:32:29.975185 12363 init.cc:377] Logged in from keytab as kudu/127.28.244.130@KRBTEST.COM (short username kudu)
I20260929 20:32:29.977063 12363 webserver.cc:460] Webserver started at http://127.28.244.130:45643/ using document root <none> and password file <none>
I20260929 20:32:29.977782 12363 fs_manager.cc:359] Metadata directory not provided
I20260929 20:32:29.977895 12363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:32:29.984169 12363 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 20:32:29.987394 12378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:29.989218 12363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:32:29.989395 12363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/wal
uuid: "8312b6a9e28542de82dc41db65e09a35"
format_stamp: "Formatted at 2026-09-29 20:32:23 on dist-test-slave-p29l"
server_key: "76df5c2f91d50add72b4b3d9dba917bf"
server_key_iv: "426b91133a80b2d5b8787c24f1163ab6"
server_key_version: "encryptionkey@0"
I20260929 20:32:29.990573 12363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:32:30.019109 12363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:32:30.024084 12363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:32:30.024410 12363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:32:30.025556 12363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:32:30.027233 12363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:32:30.027343 12363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:30.027523 12363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:32:30.027612 12363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:30.064633 12363 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.130:40679
I20260929 20:32:30.064716 12493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.130:40679 every 8 connection(s)
I20260929 20:32:30.066248 12363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
I20260929 20:32:30.071584 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 12363
I20260929 20:32:30.072175 12496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:30.069489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.130:44993)
0929 20:32:30.070486 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:30.070782 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":376,"threads_started":1}
I20260929 20:32:30.072431 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.131:40541
--local_ip_for_outbound_sockets=127.28.244.131
--tserver_master_addrs=127.28.244.190:37845
--webserver_port=41021
--webserver_interface=127.28.244.131
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:34391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:32:30.073290 12494 heartbeater.cc:644] Failed to heartbeat to 127.28.244.190:37845 (0 consecutive failures): Network error: Failed to ping master at 127.28.244.190:37845: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
I20260929 20:32:30.074458 12496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:30.073956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.130:58019)
0929 20:32:30.074173 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:30.074283 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 20:32:30.075664 12496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:30.075203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.130:53613)
0929 20:32:30.075389 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:30.075496 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 20:32:30.076056 12494 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 20:32:30.445755 12497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:32:30.446063 12497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:32:30.446214 12497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:32:30.454010 12497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:32:30.454144 12497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:32:30.454330 12497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.131
I20260929 20:32:30.464635 12497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:34391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.131:40541
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.244.131
--webserver_port=41021
--tserver_master_addrs=127.28.244.190:37845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.131
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:32:30.466301 12497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:32:30.467890 12497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:32:30.483412 12502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:30.483343 12503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:30.483367 12505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:32:30.484295 12497 server_base.cc:1034] running on GCE node
I20260929 20:32:30.484994 12497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:32:30.486085 12497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:32:30.487339 12497 hybrid_clock.cc:648] HybridClock initialized: now 1790713950487271 us; error 66 us; skew 500 ppm
I20260929 20:32:30.487702 12497 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:32:30 dist-test-slave-p29l krb5kdc[11032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713950, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:32:30.491432 12497 init.cc:377] Logged in from keytab as kudu/127.28.244.131@KRBTEST.COM (short username kudu)
I20260929 20:32:30.493312 12497 webserver.cc:460] Webserver started at http://127.28.244.131:41021/ using document root <none> and password file <none>
I20260929 20:32:30.494154 12497 fs_manager.cc:359] Metadata directory not provided
I20260929 20:32:30.494333 12497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:32:30.500944 12497 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 20:32:30.504293 12512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:30.506047 12497 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 20:32:30.506235 12497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/wal
uuid: "951e78d229d64b709e1b40963e492495"
format_stamp: "Formatted at 2026-09-29 20:32:24 on dist-test-slave-p29l"
server_key: "c450730f0350d3602a4858ebcfe55ff4"
server_key_iv: "f4e6bb5362bdbd100935e6143ad33dca"
server_key_version: "encryptionkey@0"
I20260929 20:32:30.507321 12497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:32:30.523399 12497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:32:30.527971 12497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:32:30.528223 12497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:32:30.529270 12497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:32:30.530663 12497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:32:30.530761 12497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:30.530849 12497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:32:30.530905 12497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:30.569567 12497 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.131:40541
I20260929 20:32:30.569597 12626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.131:40541 every 8 connection(s)
I20260929 20:32:30.571756 12497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
I20260929 20:32:30.578266 12629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:30.575510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.131:42153)
0929 20:32:30.576513 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:30.576837 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":443,"threads_started":1}
W20260929 20:32:30.579043 12627 heartbeater.cc:644] Failed to heartbeat to 127.28.244.190:37845 (0 consecutive failures): Network error: Failed to ping master at 127.28.244.190:37845: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
I20260929 20:32:30.580135 12629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:30.579692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.131:53035)
0929 20:32:30.579894 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:30.579979 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 20:32:30.580559 12630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:30.579665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.129:35635)
0929 20:32:30.580258 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:30.580360 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":328,"threads_started":1}
I20260929 20:32:30.580560 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 12497
I20260929 20:32:30.581681 12629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:30.581272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.131:48101)
0929 20:32:30.581457 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:30.581557 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260929 20:32:30.582048 12627 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 20:32:31.077725 12632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:31.076681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.130:46735)
0929 20:32:31.077294 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:31.077471 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":241,"threads_started":1}
I20260929 20:32:31.416435 12633 raft_consensus.cc:482] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:32:31.416836 12633 raft_consensus.cc:504] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8312b6a9e28542de82dc41db65e09a35" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40679 } } peers { permanent_uuid: "e1aa120e44bb45e68986e21a8565480e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 33535 } } peers { permanent_uuid: "951e78d229d64b709e1b40963e492495" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 40541 } }
I20260929 20:32:31.418531 12633 leader_election.cc:288] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8312b6a9e28542de82dc41db65e09a35 (127.28.244.130:40679), 951e78d229d64b709e1b40963e492495 (127.28.244.131:40541)
Sep 29 20:32:31 dist-test-slave-p29l krb5kdc[11032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713949, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for kudu/127.28.244.131@KRBTEST.COM
Sep 29 20:32:31 dist-test-slave-p29l krb5kdc[11032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713949, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for kudu/127.28.244.130@KRBTEST.COM
I20260929 20:32:31.461464 12637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:31.419292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:33093 (local address 127.28.244.131:40541)
0929 20:32:31.432945 (+ 13653us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:31.432964 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:31.433891 (+ 927us) server_negotiation.cc:407] Connection header received
0929 20:32:31.435968 (+ 2077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:31.435979 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:31.436433 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:31.436802 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:31.439233 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:31.440625 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:31.442949 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:31.443857 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:31.444073 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:31.453540 (+ 9467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:31.453551 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:31.453568 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:31.453621 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:31.457060 (+ 3439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:31.458123 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:31.458132 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:31.458138 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:31.458323 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:31.458828 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:31.458836 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:31.458839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:31.459441 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:31.459648 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:31.460599 (+ 951us) server_negotiation.cc:299] Negotiation successful
0929 20:32:31.461013 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13411,"thread_start_us":13269,"threads_started":1}
I20260929 20:32:31.461498 12634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:31.419011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:40679 (local address 127.28.244.129:47649)
0929 20:32:31.433492 (+ 14481us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:31.433559 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:31.434178 (+ 619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:31.436963 (+ 2785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:31.436974 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:31.438121 (+ 1147us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:32:31.438994 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:31.439019 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:31.440826 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:31.440830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:31.442418 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:31.442434 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:31.443776 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:31.443785 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:31.444188 (+ 403us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:31.449536 (+ 5348us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:32:31.449586 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:32:31.453227 (+ 3641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:32:31.457270 (+ 4043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:31.457282 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:31.457300 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:31.457949 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:31.458463 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:31.458468 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:31.458472 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:31.458715 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:31.459600 (+ 885us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:32:31.459608 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:32:31.460054 (+ 446us) client_negotiation.cc:769] Sending connection context
0929 20:32:31.460346 (+ 292us) client_negotiation.cc:240] Negotiation successful
0929 20:32:31.460982 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14292,"mutex_wait_us":206,"spinlock_wait_cycles":58112,"thread_start_us":14094,"threads_started":1}
I20260929 20:32:31.461613 12635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:31.419039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:40541 (local address 127.28.244.129:33093)
0929 20:32:31.433725 (+ 14686us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:31.433745 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:31.434155 (+ 410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:31.436965 (+ 2810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:31.436974 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:31.438122 (+ 1148us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:32:31.438995 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:31.439019 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:31.440829 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:31.440833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:31.442748 (+ 1915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:31.442764 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:31.444087 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:31.444092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:31.444237 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:31.449536 (+ 5299us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:32:31.449585 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:32:31.453133 (+ 3548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:32:31.457275 (+ 4142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:31.457283 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:31.457297 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:31.457950 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:31.458462 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:31.458467 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:31.458469 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:31.458706 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:31.459601 (+ 895us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:32:31.459609 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:32:31.460049 (+ 440us) client_negotiation.cc:769] Sending connection context
0929 20:32:31.460349 (+ 300us) client_negotiation.cc:240] Negotiation successful
0929 20:32:31.461156 (+ 807us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14541,"mutex_wait_us":466,"spinlock_wait_cycles":115712,"thread_start_us":14133,"threads_started":1}
I20260929 20:32:31.462797 12636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:31.419237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:47649 (local address 127.28.244.130:40679)
0929 20:32:31.432065 (+ 12828us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:31.432093 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:31.433760 (+ 1667us) server_negotiation.cc:407] Connection header received
0929 20:32:31.436051 (+ 2291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:31.436060 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:31.436434 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:31.436804 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:31.439236 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:31.440589 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:31.442699 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:31.443519 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:31.443675 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:31.453570 (+ 9895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:31.453579 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:31.453593 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:31.453638 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:31.457052 (+ 3414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:31.458128 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:31.458136 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:31.458142 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:31.458323 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:31.458835 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:31.458840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:31.458843 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:31.459444 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:31.459655 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:31.461933 (+ 2278us) server_negotiation.cc:299] Negotiation successful
0929 20:32:31.462466 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12482,"thread_start_us":12880,"threads_started":1}
W20260929 20:32:31.465760 12244 leader_election.cc:341] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 951e78d229d64b709e1b40963e492495 (127.28.244.131:40541): Not found: Tablet not found: 5110576ee03744dfb0631c5e2ef38dd0
W20260929 20:32:31.466130 12245 leader_election.cc:341] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8312b6a9e28542de82dc41db65e09a35 (127.28.244.130:40679): Not found: Tablet not found: 5110576ee03744dfb0631c5e2ef38dd0
I20260929 20:32:31.466315 12245 leader_election.cc:302] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1aa120e44bb45e68986e21a8565480e; no voters: 8312b6a9e28542de82dc41db65e09a35, 951e78d229d64b709e1b40963e492495
I20260929 20:32:31.466753 12633 raft_consensus.cc:2726] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 20:32:31.582159 12635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:31.581608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.129:55167)
0929 20:32:31.581848 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:31.581963 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260929 20:32:31.583524 12638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:31.582643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.131:37147)
0929 20:32:31.583150 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:31.583299 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":259,"threads_started":1}
I20260929 20:32:32.079900 12639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:32.078906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.130:36243)
0929 20:32:32.079579 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:32.079674 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":440,"threads_started":1}
I20260929 20:32:32.584180 12640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:32.583297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.129:57697)
0929 20:32:32.583799 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:32.583989 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":240,"threads_started":1}
I20260929 20:32:32.585341 12641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:32.584455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.131:47337)
0929 20:32:32.585073 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:32.585173 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":302,"threads_started":1}
I20260929 20:32:33.081727 12642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:33.080924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.130:37149)
0929 20:32:33.081403 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:33.081487 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":256,"threads_started":1}
I20260929 20:32:33.173558 12643 raft_consensus.cc:482] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:32:33.173858 12643 raft_consensus.cc:504] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8312b6a9e28542de82dc41db65e09a35" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40679 } } peers { permanent_uuid: "e1aa120e44bb45e68986e21a8565480e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 33535 } } peers { permanent_uuid: "951e78d229d64b709e1b40963e492495" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 40541 } }
I20260929 20:32:33.174649 12643 leader_election.cc:288] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8312b6a9e28542de82dc41db65e09a35 (127.28.244.130:40679), 951e78d229d64b709e1b40963e492495 (127.28.244.131:40541)
W20260929 20:32:33.175683 12245 leader_election.cc:341] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8312b6a9e28542de82dc41db65e09a35 (127.28.244.130:40679): Not found: Tablet not found: 5110576ee03744dfb0631c5e2ef38dd0
W20260929 20:32:33.175889 12244 leader_election.cc:341] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 951e78d229d64b709e1b40963e492495 (127.28.244.131:40541): Not found: Tablet not found: 5110576ee03744dfb0631c5e2ef38dd0
I20260929 20:32:33.176091 12244 leader_election.cc:302] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1aa120e44bb45e68986e21a8565480e; no voters: 8312b6a9e28542de82dc41db65e09a35, 951e78d229d64b709e1b40963e492495
I20260929 20:32:33.176402 12643 raft_consensus.cc:2726] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 20:32:33.586540 12644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:33.585358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.129:53555)
0929 20:32:33.586086 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:33.586255 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":297,"threads_started":1}
I20260929 20:32:33.587409 12645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:33.586427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.131:56439)
0929 20:32:33.587000 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:33.587186 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":327,"threads_started":1}
I20260929 20:32:34.084033 12646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:34.082981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.130:39699)
0929 20:32:34.083660 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:34.083771 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":414,"threads_started":1}
I20260929 20:32:34.589522 12648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:34.588696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.131:37433)
0929 20:32:34.589236 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:34.589327 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":289,"threads_started":1}
I20260929 20:32:34.589967 12647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:34.588612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.129:51935)
0929 20:32:34.589650 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:34.589774 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":899,"thread_start_us":712,"threads_started":1}
I20260929 20:32:35.086143 12649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:35.085260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.130:43851)
0929 20:32:35.085825 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:35.085924 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":291,"threads_started":1}
I20260929 20:32:35.592177 12651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:35.591035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.129:59659)
0929 20:32:35.591707 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:35.591943 (+ 236us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":322,"threads_started":1}
I20260929 20:32:35.592499 12650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:35.591015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.131:52253)
0929 20:32:35.592176 (+ 1161us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:35.592297 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1063,"thread_start_us":958,"threads_started":1}
I20260929 20:32:35.782256 12652 raft_consensus.cc:482] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:32:35.782565 12652 raft_consensus.cc:504] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8312b6a9e28542de82dc41db65e09a35" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40679 } } peers { permanent_uuid: "e1aa120e44bb45e68986e21a8565480e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 33535 } } peers { permanent_uuid: "951e78d229d64b709e1b40963e492495" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 40541 } }
I20260929 20:32:35.783334 12652 leader_election.cc:288] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8312b6a9e28542de82dc41db65e09a35 (127.28.244.130:40679), 951e78d229d64b709e1b40963e492495 (127.28.244.131:40541)
W20260929 20:32:35.784385 12244 leader_election.cc:341] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 951e78d229d64b709e1b40963e492495 (127.28.244.131:40541): Not found: Tablet not found: 5110576ee03744dfb0631c5e2ef38dd0
W20260929 20:32:35.784646 12245 leader_election.cc:341] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8312b6a9e28542de82dc41db65e09a35 (127.28.244.130:40679): Not found: Tablet not found: 5110576ee03744dfb0631c5e2ef38dd0
I20260929 20:32:35.784847 12245 leader_election.cc:302] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1aa120e44bb45e68986e21a8565480e; no voters: 8312b6a9e28542de82dc41db65e09a35, 951e78d229d64b709e1b40963e492495
I20260929 20:32:35.785182 12652 raft_consensus.cc:2726] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 20:32:36.088496 12653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:36.087453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.130:33129)
0929 20:32:36.088129 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:36.088232 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":358,"threads_started":1}
I20260929 20:32:36.594398 12654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:36.593473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.129:56367)
0929 20:32:36.594095 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:36.594197 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":372,"threads_started":1}
I20260929 20:32:36.594419 12655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:36.593568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.131:54883)
0929 20:32:36.594171 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:36.594252 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":365,"threads_started":1}
I20260929 20:32:37.090644 12656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:37.089617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.130:34565)
0929 20:32:37.090307 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:37.090415 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":359,"threads_started":1}
I20260929 20:32:37.596596 12657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:37.595541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.129:39251)
0929 20:32:37.596237 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:37.596366 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":418,"threads_started":1}
I20260929 20:32:37.596792 12658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:37.595584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.131:41129)
0929 20:32:37.596469 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:37.596563 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":762,"thread_start_us":590,"threads_started":1}
I20260929 20:32:37.936834 12659 raft_consensus.cc:482] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:32:37.937161 12659 raft_consensus.cc:504] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8312b6a9e28542de82dc41db65e09a35" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40679 } } peers { permanent_uuid: "e1aa120e44bb45e68986e21a8565480e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 33535 } } peers { permanent_uuid: "951e78d229d64b709e1b40963e492495" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 40541 } }
I20260929 20:32:37.937992 12659 leader_election.cc:288] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8312b6a9e28542de82dc41db65e09a35 (127.28.244.130:40679), 951e78d229d64b709e1b40963e492495 (127.28.244.131:40541)
W20260929 20:32:37.939064 12244 leader_election.cc:341] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 951e78d229d64b709e1b40963e492495 (127.28.244.131:40541): Not found: Tablet not found: 5110576ee03744dfb0631c5e2ef38dd0
W20260929 20:32:37.939266 12245 leader_election.cc:341] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8312b6a9e28542de82dc41db65e09a35 (127.28.244.130:40679): Not found: Tablet not found: 5110576ee03744dfb0631c5e2ef38dd0
I20260929 20:32:37.939446 12245 leader_election.cc:302] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1aa120e44bb45e68986e21a8565480e; no voters: 8312b6a9e28542de82dc41db65e09a35, 951e78d229d64b709e1b40963e492495
I20260929 20:32:37.939779 12659 raft_consensus.cc:2726] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 20:32:38.093120 12660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:38.092328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.130:44937)
0929 20:32:38.092830 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:38.092931 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":256,"threads_started":1}
I20260929 20:32:38.598802 12661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:38.597799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.131:49113)
0929 20:32:38.598494 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:38.598590 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":491,"threads_started":1}
I20260929 20:32:38.598826 12662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:38.597819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.129:39363)
0929 20:32:38.598480 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:38.598600 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":329,"threads_started":1}
I20260929 20:32:39.095242 12663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:39.094229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.130:55279)
0929 20:32:39.094885 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:39.095051 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":272,"threads_started":1}
I20260929 20:32:39.600991 12664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:39.600091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.129:41543)
0929 20:32:39.600672 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:39.600805 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":321,"threads_started":1}
I20260929 20:32:39.601083 12665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:39.600224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.131:57301)
0929 20:32:39.600824 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:39.600906 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":272,"threads_started":1}
I20260929 20:32:40.097219 12666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:40.096397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.130:59089)
0929 20:32:40.096925 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:40.097017 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":291,"threads_started":1}
I20260929 20:32:40.603188 12667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:40.602069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.129:36871)
0929 20:32:40.602821 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:40.602943 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":477,"threads_started":1}
I20260929 20:32:40.603258 12668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:40.602211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.131:51951)
0929 20:32:40.602965 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:40.603055 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":528,"threads_started":1}
I20260929 20:32:41.099143 12669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:41.098295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.130:44085)
0929 20:32:41.098865 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:41.098959 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":258,"threads_started":1}
I20260929 20:32:41.605366 12671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:41.604304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.129:42253)
0929 20:32:41.604921 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:41.605084 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":285,"threads_started":1}
I20260929 20:32:41.606302 12670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:41.604301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.131:56779)
0929 20:32:41.606063 (+ 1762us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:41.606150 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1649,"thread_start_us":1471,"threads_started":1}
I20260929 20:32:42.101296 12672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:42.100302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.130:40533)
0929 20:32:42.101016 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:42.101105 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":481,"threads_started":1}
I20260929 20:32:42.607955 12673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:42.606636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.129:45689)
0929 20:32:42.607588 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:42.607697 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":821,"thread_start_us":677,"threads_started":1}
I20260929 20:32:42.608167 12674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:42.607358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.131:53659)
0929 20:32:42.607875 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:42.607951 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":273,"threads_started":1}
I20260929 20:32:43.103317 12675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:43.102416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.130:50097)
0929 20:32:43.103024 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:43.103132 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":334,"threads_started":1}
I20260929 20:32:43.609929 12676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:43.609070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.129:52185)
0929 20:32:43.609627 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:43.609742 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":310,"threads_started":1}
I20260929 20:32:43.610205 12677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:43.609235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.131:47665)
0929 20:32:43.609850 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:43.609950 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":336,"threads_started":1}
I20260929 20:32:43.629522 12678 raft_consensus.cc:482] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:32:43.629778 12678 raft_consensus.cc:504] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8312b6a9e28542de82dc41db65e09a35" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40679 } } peers { permanent_uuid: "e1aa120e44bb45e68986e21a8565480e" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 33535 } } peers { permanent_uuid: "951e78d229d64b709e1b40963e492495" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 40541 } }
I20260929 20:32:43.630530 12678 leader_election.cc:288] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8312b6a9e28542de82dc41db65e09a35 (127.28.244.130:40679), 951e78d229d64b709e1b40963e492495 (127.28.244.131:40541)
W20260929 20:32:43.631502 12245 leader_election.cc:341] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8312b6a9e28542de82dc41db65e09a35 (127.28.244.130:40679): Not found: Tablet not found: 5110576ee03744dfb0631c5e2ef38dd0
W20260929 20:32:43.631743 12244 leader_election.cc:341] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 951e78d229d64b709e1b40963e492495 (127.28.244.131:40541): Not found: Tablet not found: 5110576ee03744dfb0631c5e2ef38dd0
I20260929 20:32:43.631968 12244 leader_election.cc:302] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1aa120e44bb45e68986e21a8565480e; no voters: 8312b6a9e28542de82dc41db65e09a35, 951e78d229d64b709e1b40963e492495
I20260929 20:32:43.632272 12678 raft_consensus.cc:2726] T 5110576ee03744dfb0631c5e2ef38dd0 P e1aa120e44bb45e68986e21a8565480e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 20:32:44.105429 12679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:44.104517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.130:34637)
0929 20:32:44.105099 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:44.105213 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":281,"threads_started":1}
I20260929 20:32:44.612473 12681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:44.611397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.129:51203)
0929 20:32:44.612020 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:44.612186 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":309,"threads_started":1}
I20260929 20:32:44.612782 12680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:44.611331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.131:52051)
0929 20:32:44.612346 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:44.612507 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":309,"threads_started":1}
I20260929 20:32:45.107757 12682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:45.106847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.130:37213)
0929 20:32:45.107409 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:45.107540 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":296,"threads_started":1}
I20260929 20:32:45.593798 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 12193
I20260929 20:32:45.614859 12684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:45.613720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.129:54319)
0929 20:32:45.614425 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:45.614576 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":355,"threads_started":1}
I20260929 20:32:45.615103 12685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:45.614307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.131:33275)
0929 20:32:45.614765 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:45.614885 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":246,"threads_started":1}
I20260929 20:32:46.109851 12686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:46.108924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.130:40701)
0929 20:32:46.109536 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:46.109638 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":306,"threads_started":1}
I20260929 20:32:46.536242 12683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:45.595252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41674 (local address 127.28.244.129:33535)
0929 20:32:45.596398 (+ 1146us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:45.596405 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:46.520684 (+924279us) server_negotiation.cc:407] Connection header received
0929 20:32:46.520859 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:46.520867 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:46.521124 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:46.521305 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:46.522990 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:46.524138 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:46.525654 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:46.526294 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:46.526409 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:46.529608 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:46.529624 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:46.529640 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:46.529683 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:46.532917 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:46.533813 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:46.533822 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:46.533824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:46.533994 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:46.534441 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:46.534445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:46.534447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:46.534893 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:46.535149 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:46.535502 (+ 353us) server_negotiation.cc:299] Negotiation successful
0929 20:32:46.535760 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":950,"thread_start_us":771,"threads_started":1}
I20260929 20:32:46.537669 12354 generic_service.cc:196] Checking for leaks (request via RPC)
W20260929 20:32:46.595486 29650 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 12193: Timed out: CheckLeaks RPC to 127.28.244.129:33535 timed out after 1.000s (SENT)
I20260929 20:32:46.595763 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 12193
I20260929 20:32:46.612972 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 12363
I20260929 20:32:46.617061 12689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:46.616224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:37845 (local address 127.28.244.131:42343)
0929 20:32:46.616790 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:46.616880 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":293,"threads_started":1}
I20260929 20:32:46.629823 12688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:46.613986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37806 (local address 127.28.244.130:40679)
0929 20:32:46.614475 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:46.614479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:46.614498 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:32:46.614559 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:46.614562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:46.614634 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:46.614747 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:46.616442 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:46.617570 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:46.619025 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:46.619486 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:46.619599 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:46.622789 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:46.622796 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:46.622800 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:46.622853 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:46.626586 (+ 3733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:46.627526 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:46.627531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:46.627533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:46.627681 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:46.628158 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:46.628165 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:46.628171 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:46.628509 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:46.628822 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:46.629163 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 20:32:46.629421 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":274,"threads_started":1}
I20260929 20:32:46.630564 12489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:32:46.717823 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 12363
I20260929 20:32:46.737133 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 12497
I20260929 20:32:46.753219 12691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:46.738227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33736 (local address 127.28.244.131:40541)
0929 20:32:46.738963 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:46.738968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:46.738986 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:32:46.739047 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:46.739051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:46.739124 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:46.739236 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:46.740908 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:46.741842 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:46.743300 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:46.743787 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:46.744101 (+ 314us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:46.747271 (+ 3170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:46.747277 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:46.747281 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:46.747325 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:46.750369 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:46.751249 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:46.751253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:46.751256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:46.751412 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:46.751765 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:46.751769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:46.751771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:46.752146 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:46.752379 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:46.752656 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 20:32:46.752886 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":407,"threads_started":1}
I20260929 20:32:46.753994 12622 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:32:46.842615 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 12497
2026-09-29T20:32:46Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (26388 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:32:46 dist-test-slave-p29l krb5kdc[12696](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:32:46 dist-test-slave-p29l krb5kdc[12696](info): set up 2 sockets
Sep 29 20:32:46 dist-test-slave-p29l krb5kdc[12696](info): commencing operation
krb5kdc: starting...
W20260929 20:32:47.996129 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:32:48 dist-test-slave-p29l krb5kdc[12696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:32:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:32:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.244.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
I20260929 20:32:48.270339 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:35929
--webserver_interface=127.28.244.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:35929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:32:48.651065 12709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:32:48.651484 12709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:32:48.651666 12709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:32:48.659586 12709 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 20:32:48.659796 12709 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 20:32:48.659884 12709 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 20:32:48.659988 12709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:32:48.660092 12709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:32:48.660200 12709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:32:48.660284 12709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:32:48.660363 12709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:32:48.672106 12709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:35929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:35929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:32:48.673571 12709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:32:48.675066 12709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:32:48.688599 12714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:48.688599 12715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:48.688699 12717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:32:48.689684 12709 server_base.cc:1034] running on GCE node
I20260929 20:32:48.690551 12709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:32:48.692238 12709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:32:48.693509 12709 hybrid_clock.cc:648] HybridClock initialized: now 1790713968693414 us; error 65 us; skew 500 ppm
I20260929 20:32:48.693912 12709 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:32:48 dist-test-slave-p29l krb5kdc[12696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713968, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:32:48.697801 12709 init.cc:377] Logged in from keytab as kudu/127.28.244.190@KRBTEST.COM (short username kudu)
I20260929 20:32:48.699566 12709 webserver.cc:460] Webserver started at http://127.28.244.190:34105/ using document root <none> and password file <none>
I20260929 20:32:48.700439 12709 fs_manager.cc:359] Metadata directory not provided
I20260929 20:32:48.700590 12709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:32:48.700949 12709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:32:48.704365 12709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "391fcaa2e0d84f20b5f803c250eea457"
format_stamp: "Formatted at 2026-09-29 20:32:48 on dist-test-slave-p29l"
server_key: "4b21e2e5d948004ed4c01ce708b1f639"
server_key_iv: "3ab699e47e976be32682fc4d568d3e32"
server_key_version: "encryptionkey@0"
I20260929 20:32:48.705284 12709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "391fcaa2e0d84f20b5f803c250eea457"
format_stamp: "Formatted at 2026-09-29 20:32:48 on dist-test-slave-p29l"
server_key: "4b21e2e5d948004ed4c01ce708b1f639"
server_key_iv: "3ab699e47e976be32682fc4d568d3e32"
server_key_version: "encryptionkey@0"
I20260929 20:32:48.712306 12709 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260929 20:32:48.716307 12724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:48.718143 12709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:32:48.718396 12709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "391fcaa2e0d84f20b5f803c250eea457"
format_stamp: "Formatted at 2026-09-29 20:32:48 on dist-test-slave-p29l"
server_key: "4b21e2e5d948004ed4c01ce708b1f639"
server_key_iv: "3ab699e47e976be32682fc4d568d3e32"
server_key_version: "encryptionkey@0"
I20260929 20:32:48.718704 12709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:32:48.739264 12709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:32:48.744410 12709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:32:48.744756 12709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:32:48.767112 12709 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:35929
I20260929 20:32:48.767175 12776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:35929 every 8 connection(s)
I20260929 20:32:48.768725 12709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:32:48.773059 12777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:32:48.778967 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 12709
I20260929 20:32:48.779223 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:32:48.779958 29650 external_mini_cluster.cc:1428] Setting key 610bc8cff3622a64feea36cd229bdc13
I20260929 20:32:48.786262 12777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457: Bootstrap starting.
I20260929 20:32:48.790491 12777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457: Neither blocks nor log segments found. Creating new log.
I20260929 20:32:48.792018 12777 log.cc:824] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457: Log is configured to *not* fsync() on all Append() calls
I20260929 20:32:48.795878 12777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457: No bootstrap required, opened a new log
Sep 29 20:32:48 dist-test-slave-p29l krb5kdc[12696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:32:48.807861 12777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391fcaa2e0d84f20b5f803c250eea457" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 35929 } }
I20260929 20:32:48.808394 12777 raft_consensus.cc:374] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:32:48.808501 12777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 391fcaa2e0d84f20b5f803c250eea457, State: Initialized, Role: FOLLOWER
I20260929 20:32:48.809237 12777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391fcaa2e0d84f20b5f803c250eea457" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 35929 } }
I20260929 20:32:48.809577 12777 raft_consensus.cc:388] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:32:48.809613 12780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:48.782227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58470 (local address 127.28.244.190:35929)
0929 20:32:48.783242 (+ 1015us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:48.783266 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:48.783322 (+ 56us) server_negotiation.cc:407] Connection header received
0929 20:32:48.785871 (+ 2549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:48.785905 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:48.786494 (+ 589us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:48.786989 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:48.788606 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:48.790649 (+ 2043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:48.792952 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:48.793860 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:48.794076 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:48.798934 (+ 4858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:48.798946 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:48.798972 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:48.799045 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:48.803716 (+ 4671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:48.804849 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:48.804864 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:48.804878 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:48.805171 (+ 293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:48.805804 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:48.805809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:48.805813 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:48.806718 (+ 905us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:48.807044 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:48.807556 (+ 512us) server_negotiation.cc:299] Negotiation successful
0929 20:32:48.808121 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":733,"thread_start_us":467,"threads_started":1}
I20260929 20:32:48.809751 12777 raft_consensus.cc:482] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:32:48.809993 12777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:32:48.812773 12777 raft_consensus.cc:504] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391fcaa2e0d84f20b5f803c250eea457" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 35929 } }
I20260929 20:32:48.813462 12777 leader_election.cc:302] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 391fcaa2e0d84f20b5f803c250eea457; no voters:
I20260929 20:32:48.814180 12777 leader_election.cc:288] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:32:48.814320 12782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:32:48.815796 12782 raft_consensus.cc:686] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [term 1 LEADER]: Becoming Leader. State: Replica: 391fcaa2e0d84f20b5f803c250eea457, State: Running, Role: LEADER
I20260929 20:32:48.816385 12782 consensus_queue.cc:237] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391fcaa2e0d84f20b5f803c250eea457" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 35929 } }
I20260929 20:32:48.816692 12777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:32:48.820112 12784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 391fcaa2e0d84f20b5f803c250eea457. Latest consensus state: current_term: 1 leader_uuid: "391fcaa2e0d84f20b5f803c250eea457" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391fcaa2e0d84f20b5f803c250eea457" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 35929 } } }
I20260929 20:32:48.820173 12783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "391fcaa2e0d84f20b5f803c250eea457" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391fcaa2e0d84f20b5f803c250eea457" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 35929 } } }
I20260929 20:32:48.820617 12784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [sys.catalog]: This master's current role is: LEADER
I20260929 20:32:48.820627 12783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [sys.catalog]: This master's current role is: LEADER
I20260929 20:32:48.829134 12792 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:32:48.834097 12792 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:32:48.844537 12792 catalog_manager.cc:1347] Generated new cluster ID: bdece67bdd2a4aacbc18219848e5f87a
I20260929 20:32:48.844687 12792 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:32:48.853840 12792 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:32:48.855396 12792 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:32:48.866575 12792 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457: Generated new TSK 0
I20260929 20:32:48.867957 12792 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.244.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab.
Entry for principal kudu/127.28.244.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab.
I20260929 20:32:48.941375 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.129:0
--local_ip_for_outbound_sockets=127.28.244.129
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:35929
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:32:49.309448 12803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:32:49.309778 12803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:32:49.309875 12803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:32:49.319031 12803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:32:49.319175 12803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:32:49.319306 12803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.129
I20260929 20:32:49.332840 12803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:35929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.129
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:32:49.334287 12803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:32:49.335872 12803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:32:49.351655 12809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:49.351701 12811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:49.351734 12808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:32:49.352639 12803 server_base.cc:1034] running on GCE node
I20260929 20:32:49.353364 12803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:32:49.354511 12803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:32:49.355736 12803 hybrid_clock.cc:648] HybridClock initialized: now 1790713969355660 us; error 61 us; skew 500 ppm
I20260929 20:32:49.356230 12803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:32:49 dist-test-slave-p29l krb5kdc[12696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713969, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:32:49.363369 12803 init.cc:377] Logged in from keytab as kudu/127.28.244.129@KRBTEST.COM (short username kudu)
I20260929 20:32:49.365187 12803 webserver.cc:460] Webserver started at http://127.28.244.129:43257/ using document root <none> and password file <none>
I20260929 20:32:49.366169 12803 fs_manager.cc:359] Metadata directory not provided
I20260929 20:32:49.366365 12803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:32:49.366741 12803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:32:49.370270 12803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/data/instance:
uuid: "64602944447a488d8a0ac58f14700eb6"
format_stamp: "Formatted at 2026-09-29 20:32:49 on dist-test-slave-p29l"
server_key: "971a8af187bd30975ba0755262c6c45d"
server_key_iv: "375b1677fced77eb8228f478c9e3bf40"
server_key_version: "encryptionkey@0"
I20260929 20:32:49.371194 12803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance:
uuid: "64602944447a488d8a0ac58f14700eb6"
format_stamp: "Formatted at 2026-09-29 20:32:49 on dist-test-slave-p29l"
server_key: "971a8af187bd30975ba0755262c6c45d"
server_key_iv: "375b1677fced77eb8228f478c9e3bf40"
server_key_version: "encryptionkey@0"
I20260929 20:32:49.377434 12803 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260929 20:32:49.381346 12818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:49.382989 12803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:32:49.383212 12803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/wal
uuid: "64602944447a488d8a0ac58f14700eb6"
format_stamp: "Formatted at 2026-09-29 20:32:49 on dist-test-slave-p29l"
server_key: "971a8af187bd30975ba0755262c6c45d"
server_key_iv: "375b1677fced77eb8228f478c9e3bf40"
server_key_version: "encryptionkey@0"
I20260929 20:32:49.383445 12803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:32:49.414605 12803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:32:49.419771 12803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:32:49.420181 12803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:32:49.421329 12803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:32:49.423003 12803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:32:49.423116 12803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:49.423301 12803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:32:49.423403 12803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:49.460291 12803 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.129:42005
I20260929 20:32:49.460387 12932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.129:42005 every 8 connection(s)
I20260929 20:32:49.461768 12803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
I20260929 20:32:49.468783 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 12803
I20260929 20:32:49.469038 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance
I20260929 20:32:49.469836 29650 external_mini_cluster.cc:1428] Setting key bd30a0dbad971abd718a5f7848ecee77
Sep 29 20:32:49 dist-test-slave-p29l krb5kdc[12696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713969, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:32:49.484565 12936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:49.464869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:37821 (local address 127.28.244.190:35929)
0929 20:32:49.465374 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:49.465379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:49.466322 (+ 943us) server_negotiation.cc:407] Connection header received
0929 20:32:49.468350 (+ 2028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:49.468356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:49.468456 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:49.468585 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:49.470471 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:49.471381 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:49.473279 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:49.473713 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:49.473872 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:49.477864 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:49.477869 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:49.477873 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:49.477911 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:49.480730 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:49.481635 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:49.481640 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:49.481643 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:49.481785 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:49.482278 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:49.482285 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:49.482292 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:49.482580 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:49.482748 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:49.483711 (+ 963us) server_negotiation.cc:299] Negotiation successful
0929 20:32:49.484057 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":281,"threads_started":1}
I20260929 20:32:49.485456 12937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:49.465406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.129:37821)
0929 20:32:49.466125 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:49.466164 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:49.468040 (+ 1876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:49.468878 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:49.468893 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:49.469566 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:32:49.470254 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:49.470280 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:49.471579 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:49.471583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:49.473108 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:49.473123 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:49.473995 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:49.474003 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:49.474275 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:49.475046 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:32:49.475070 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:32:49.477573 (+ 2503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:32:49.480941 (+ 3368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:49.480952 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:49.480979 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:49.481476 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:49.481910 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:49.481915 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:49.481919 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:49.482145 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:49.482723 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:32:49.482732 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:32:49.483137 (+ 405us) client_negotiation.cc:769] Sending connection context
0929 20:32:49.483486 (+ 349us) client_negotiation.cc:240] Negotiation successful
0929 20:32:49.484061 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":283,"threads_started":1}
I20260929 20:32:49.487320 12933 heartbeater.cc:347] Connected to a master server at 127.28.244.190:35929
I20260929 20:32:49.487717 12933 heartbeater.cc:464] Registering TS with master...
I20260929 20:32:49.488751 12933 heartbeater.cc:511] Master 127.28.244.190:35929 requested a full tablet report, sending...
I20260929 20:32:49.491534 12742 ts_manager.cc:194] Registered new tserver with Master: 64602944447a488d8a0ac58f14700eb6 (127.28.244.129:42005)
I20260929 20:32:49.494010 12742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.129@KRBTEST.COM'} at 127.28.244.129:37821
WARNING: no policy specified for kudu/127.28.244.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab.
Entry for principal kudu/127.28.244.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab.
I20260929 20:32:49.527750 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.130:0
--local_ip_for_outbound_sockets=127.28.244.130
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:35929
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:32:49.900995 12940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:32:49.901379 12940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:32:49.901564 12940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:32:49.909519 12940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:32:49.909725 12940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:32:49.909924 12940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.130
I20260929 20:32:49.921216 12940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:35929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.130
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:32:49.922747 12940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:32:49.924402 12940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:32:49.939882 12945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:49.939917 12946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:49.939990 12948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:32:49.940457 12940 server_base.cc:1034] running on GCE node
I20260929 20:32:49.941264 12940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:32:49.942471 12940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:32:49.943722 12940 hybrid_clock.cc:648] HybridClock initialized: now 1790713969943667 us; error 59 us; skew 500 ppm
I20260929 20:32:49.944114 12940 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:32:49 dist-test-slave-p29l krb5kdc[12696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713969, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:32:49.947774 12940 init.cc:377] Logged in from keytab as kudu/127.28.244.130@KRBTEST.COM (short username kudu)
I20260929 20:32:49.949529 12940 webserver.cc:460] Webserver started at http://127.28.244.130:42909/ using document root <none> and password file <none>
I20260929 20:32:49.950328 12940 fs_manager.cc:359] Metadata directory not provided
I20260929 20:32:49.950495 12940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:32:49.950832 12940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:32:49.954808 12940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/data/instance:
uuid: "e2ed5f2c06fb4efdb4f3e9de12ed66fb"
format_stamp: "Formatted at 2026-09-29 20:32:49 on dist-test-slave-p29l"
server_key: "aa0c4959742f053c7875eb4cebcb0a40"
server_key_iv: "f208ac299da3e23631a51bbe7394b0f4"
server_key_version: "encryptionkey@0"
I20260929 20:32:49.955920 12940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance:
uuid: "e2ed5f2c06fb4efdb4f3e9de12ed66fb"
format_stamp: "Formatted at 2026-09-29 20:32:49 on dist-test-slave-p29l"
server_key: "aa0c4959742f053c7875eb4cebcb0a40"
server_key_iv: "f208ac299da3e23631a51bbe7394b0f4"
server_key_version: "encryptionkey@0"
I20260929 20:32:49.962549 12940 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 20:32:49.967208 12955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:49.968888 12940 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 20:32:49.969120 12940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/wal
uuid: "e2ed5f2c06fb4efdb4f3e9de12ed66fb"
format_stamp: "Formatted at 2026-09-29 20:32:49 on dist-test-slave-p29l"
server_key: "aa0c4959742f053c7875eb4cebcb0a40"
server_key_iv: "f208ac299da3e23631a51bbe7394b0f4"
server_key_version: "encryptionkey@0"
I20260929 20:32:49.969427 12940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:32:49.994818 12940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:32:50.000236 12940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:32:50.000567 12940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:32:50.001695 12940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:32:50.003189 12940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:32:50.003299 12940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:50.003450 12940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:32:50.003521 12940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:50.039721 12940 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.130:33195
I20260929 20:32:50.039783 13069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.130:33195 every 8 connection(s)
I20260929 20:32:50.041289 12940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
I20260929 20:32:50.046093 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 12940
I20260929 20:32:50.046306 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance
I20260929 20:32:50.046948 29650 external_mini_cluster.cc:1428] Setting key 802663735e052f16525fc166c1e1206a
Sep 29 20:32:50 dist-test-slave-p29l krb5kdc[12696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713969, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:32:50.067763 13072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:50.044093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:51885 (local address 127.28.244.190:35929)
0929 20:32:50.044592 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:50.044599 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:50.045696 (+ 1097us) server_negotiation.cc:407] Connection header received
0929 20:32:50.048321 (+ 2625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:50.048330 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:50.048426 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:50.048523 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:50.051023 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:50.051963 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:50.054754 (+ 2791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:50.055259 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:50.055389 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:50.060456 (+ 5067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:50.060462 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:50.060465 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:50.060509 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:50.063519 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:50.064789 (+ 1270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:50.064796 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:50.064798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:50.064957 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:50.065612 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:50.065617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:50.065619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:50.065971 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:50.066125 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:50.067195 (+ 1070us) server_negotiation.cc:299] Negotiation successful
0929 20:32:50.067438 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":236,"threads_started":1}
I20260929 20:32:50.069314 13073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:50.044699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.130:51885)
0929 20:32:50.045460 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:50.045508 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:50.048023 (+ 2515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:50.048870 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:50.048883 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:50.049731 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:32:50.050761 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:50.050796 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:50.052149 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:50.052158 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:50.054477 (+ 2319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:50.054499 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:50.055554 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:50.055564 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:50.055977 (+ 413us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:50.056843 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:32:50.056882 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:32:50.060092 (+ 3210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:32:50.063748 (+ 3656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:50.063762 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:50.063787 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:50.064588 (+ 801us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:50.065111 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:50.065119 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:50.065126 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:50.065418 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:50.066133 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:32:50.066145 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:32:50.066700 (+ 555us) client_negotiation.cc:769] Sending connection context
0929 20:32:50.067052 (+ 352us) client_negotiation.cc:240] Negotiation successful
0929 20:32:50.067701 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":278,"threads_started":1}
I20260929 20:32:50.071298 13070 heartbeater.cc:347] Connected to a master server at 127.28.244.190:35929
I20260929 20:32:50.071666 13070 heartbeater.cc:464] Registering TS with master...
I20260929 20:32:50.072546 13070 heartbeater.cc:511] Master 127.28.244.190:35929 requested a full tablet report, sending...
I20260929 20:32:50.074362 12742 ts_manager.cc:194] Registered new tserver with Master: e2ed5f2c06fb4efdb4f3e9de12ed66fb (127.28.244.130:33195)
WARNING: no policy specified for kudu/127.28.244.131@KRBTEST.COM; defaulting to no policy
I20260929 20:32:50.075764 12742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.130@KRBTEST.COM'} at 127.28.244.130:51885
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab.
Entry for principal kudu/127.28.244.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab.
I20260929 20:32:50.112028 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.131:0
--local_ip_for_outbound_sockets=127.28.244.131
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:35929
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:32:50.482885 13076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:32:50.483207 13076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:32:50.483381 13076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:32:50.491276 13076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:32:50.491408 13076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:32:50.491593 13076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.131
I20260929 20:32:50.498102 12933 heartbeater.cc:503] Master 127.28.244.190:35929 was elected leader, sending a full tablet report...
I20260929 20:32:50.502923 13076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:35929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.131
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:32:50.504474 13076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:32:50.505891 13076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:32:50.520309 13081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:50.520372 13085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:32:50.520349 13082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:32:50.521315 13076 server_base.cc:1034] running on GCE node
I20260929 20:32:50.522050 13076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:32:50.523221 13076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:32:50.524507 13076 hybrid_clock.cc:648] HybridClock initialized: now 1790713970524424 us; error 62 us; skew 500 ppm
I20260929 20:32:50.524899 13076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:32:50 dist-test-slave-p29l krb5kdc[12696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713970, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:32:50.528292 13076 init.cc:377] Logged in from keytab as kudu/127.28.244.131@KRBTEST.COM (short username kudu)
I20260929 20:32:50.529906 13076 webserver.cc:460] Webserver started at http://127.28.244.131:45243/ using document root <none> and password file <none>
I20260929 20:32:50.530632 13076 fs_manager.cc:359] Metadata directory not provided
I20260929 20:32:50.530753 13076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:32:50.531019 13076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:32:50.534369 13076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/data/instance:
uuid: "6a86a4b292a840b58fc773f99295c67c"
format_stamp: "Formatted at 2026-09-29 20:32:50 on dist-test-slave-p29l"
server_key: "3324456691340329677eae6bf32be348"
server_key_iv: "79e5be5939164f1c12c25b5a58dda085"
server_key_version: "encryptionkey@0"
I20260929 20:32:50.535205 13076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance:
uuid: "6a86a4b292a840b58fc773f99295c67c"
format_stamp: "Formatted at 2026-09-29 20:32:50 on dist-test-slave-p29l"
server_key: "3324456691340329677eae6bf32be348"
server_key_iv: "79e5be5939164f1c12c25b5a58dda085"
server_key_version: "encryptionkey@0"
I20260929 20:32:50.541381 13076 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260929 20:32:50.545578 13092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:50.547292 13076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260929 20:32:50.547521 13076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/wal
uuid: "6a86a4b292a840b58fc773f99295c67c"
format_stamp: "Formatted at 2026-09-29 20:32:50 on dist-test-slave-p29l"
server_key: "3324456691340329677eae6bf32be348"
server_key_iv: "79e5be5939164f1c12c25b5a58dda085"
server_key_version: "encryptionkey@0"
I20260929 20:32:50.547760 13076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:32:50.568302 13076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:32:50.573325 13076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:32:50.573594 13076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:32:50.574796 13076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:32:50.576457 13076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:32:50.576546 13076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:50.576640 13076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:32:50.576685 13076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:32:50.611629 13076 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.131:36139
I20260929 20:32:50.611755 13204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.131:36139 every 8 connection(s)
I20260929 20:32:50.613068 13076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
I20260929 20:32:50.619849 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 13076
I20260929 20:32:50.620105 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance
I20260929 20:32:50.620745 29650 external_mini_cluster.cc:1428] Setting key 190e6f4cbb1e29034d548441d901c962
Sep 29 20:32:50 dist-test-slave-p29l krb5kdc[12696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713970, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:32:50.634935 13207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:50.615989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:57355 (local address 127.28.244.190:35929)
0929 20:32:50.616464 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:50.616469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:50.617412 (+ 943us) server_negotiation.cc:407] Connection header received
0929 20:32:50.619414 (+ 2002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:50.619420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:50.619515 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:50.619671 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:50.621622 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:50.622462 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:50.624400 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:50.624843 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:50.624988 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:50.628714 (+ 3726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:50.628719 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:50.628722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:50.628767 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:50.631469 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:50.632381 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:50.632385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:50.632388 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:50.632526 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:50.633024 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:50.633028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:50.633030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:50.633324 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:50.633495 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:50.634265 (+ 770us) server_negotiation.cc:299] Negotiation successful
0929 20:32:50.634514 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":241,"threads_started":1}
I20260929 20:32:50.636157 13208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:50.616438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.131:57355)
0929 20:32:50.617223 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:50.617287 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:50.619093 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:50.620024 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:50.620037 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:50.620735 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:32:50.621425 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:50.621446 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:50.622660 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:50.622665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:50.624231 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:50.624252 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:50.625090 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:50.625100 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:50.625352 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:50.626133 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:32:50.626163 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:32:50.628439 (+ 2276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:32:50.631641 (+ 3202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:50.631652 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:50.631674 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:50.632223 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:50.632665 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:32:50.632670 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:32:50.632674 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:32:50.632891 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:32:50.633464 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:32:50.633473 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:32:50.633878 (+ 405us) client_negotiation.cc:769] Sending connection context
0929 20:32:50.634167 (+ 289us) client_negotiation.cc:240] Negotiation successful
0929 20:32:50.634753 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":296,"threads_started":1}
I20260929 20:32:50.637593 13205 heartbeater.cc:347] Connected to a master server at 127.28.244.190:35929
I20260929 20:32:50.637974 13205 heartbeater.cc:464] Registering TS with master...
I20260929 20:32:50.638870 13205 heartbeater.cc:511] Master 127.28.244.190:35929 requested a full tablet report, sending...
I20260929 20:32:50.640621 12742 ts_manager.cc:194] Registered new tserver with Master: 6a86a4b292a840b58fc773f99295c67c (127.28.244.131:36139)
I20260929 20:32:50.641530 12742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.131@KRBTEST.COM'} at 127.28.244.131:57355
I20260929 20:32:50.650064 29650 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:32:50.669734 13207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:50.655356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58476 (local address 127.28.244.190:35929)
0929 20:32:50.655628 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:50.655637 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:50.656068 (+ 431us) server_negotiation.cc:407] Connection header received
0929 20:32:50.656243 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:50.656248 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:50.656325 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:50.656416 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:50.658002 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:50.658794 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:50.660404 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:50.660826 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:50.660931 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:50.663554 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:50.663559 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:50.663563 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:50.663607 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:50.666549 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:50.667358 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:50.667364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:50.667366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:50.667510 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:50.668070 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:50.668075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:50.668077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:50.668484 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:50.668688 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:50.669147 (+ 459us) server_negotiation.cc:299] Negotiation successful
0929 20:32:50.669352 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 20:32:50.674481 12742 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58476:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:32:50.682548 12742 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:32:50.719717 13221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:50.697764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52014 (local address 127.28.244.130:33195)
0929 20:32:50.710644 (+ 12880us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:50.710656 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:50.710697 (+ 41us) server_negotiation.cc:407] Connection header received
0929 20:32:50.710831 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:50.710842 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:50.711157 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:50.711386 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:32:50.712165 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:50.714098 (+ 1933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:50.716782 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:50.718343 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:50.718560 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:50.718764 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:50.719116 (+ 352us) server_negotiation.cc:299] Negotiation successful
0929 20:32:50.719322 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12706,"thread_start_us":12581,"threads_started":1}
I20260929 20:32:50.719775 13220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:50.697239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:33195 (local address 127.0.0.1:52014)
0929 20:32:50.709086 (+ 11847us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:50.709162 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:50.709455 (+ 293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:50.711429 (+ 1974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:50.711436 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:50.711480 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:32:50.711959 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:50.711979 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:50.714273 (+ 2294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:50.714278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:50.716561 (+ 2283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:50.716578 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:50.718561 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:50.718568 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:50.718812 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:50.718877 (+ 65us) client_negotiation.cc:769] Sending connection context
0929 20:32:50.719022 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 20:32:50.719264 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11445,"mutex_wait_us":109,"thread_start_us":13482,"threads_started":1}
I20260929 20:32:50.721652 13223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:50.697714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:42005 (local address 127.0.0.1:60878)
0929 20:32:50.710830 (+ 13116us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:50.710865 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:50.711042 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:50.711824 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:50.711830 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:50.711859 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:32:50.712765 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:50.712784 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:50.715695 (+ 2911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:50.715703 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:50.718272 (+ 2569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:50.718289 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:50.720544 (+ 2255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:50.720557 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:50.720823 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:50.720879 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 20:32:50.720965 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 20:32:50.721117 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12942,"mutex_wait_us":53,"thread_start_us":10762,"threads_started":1}
I20260929 20:32:50.721715 13219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:50.698058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:36139 (local address 127.0.0.1:46168)
0929 20:32:50.711078 (+ 13020us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:50.711099 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:50.711237 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:50.712141 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:50.712149 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:50.712177 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:32:50.712858 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:50.712874 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:50.715269 (+ 2395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:50.715275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:50.717718 (+ 2443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:50.717736 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:50.720479 (+ 2743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:50.720484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:50.720921 (+ 437us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:50.720972 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 20:32:50.721056 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 20:32:50.721208 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12893,"mutex_wait_us":118,"thread_start_us":12419,"threads_started":1}
I20260929 20:32:50.722536 13224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:50.698388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60878 (local address 127.28.244.129:42005)
0929 20:32:50.707358 (+ 8970us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:50.707373 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:50.711011 (+ 3638us) server_negotiation.cc:407] Connection header received
0929 20:32:50.711256 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:50.711264 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:50.711560 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:50.711793 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:32:50.713026 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:50.715540 (+ 2514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:50.718475 (+ 2935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:50.719992 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:50.720203 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:50.720375 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:50.721916 (+ 1541us) server_negotiation.cc:299] Negotiation successful
0929 20:32:50.722177 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8687,"thread_start_us":8452,"threads_started":1}
I20260929 20:32:50.722594 13222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:50.697685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46168 (local address 127.28.244.131:36139)
0929 20:32:50.710264 (+ 12579us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:50.710278 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:50.711258 (+ 980us) server_negotiation.cc:407] Connection header received
0929 20:32:50.711369 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:50.711376 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:50.711597 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:50.711793 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:32:50.713101 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:50.715086 (+ 1985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:50.717989 (+ 2903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:50.719794 (+ 1805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:50.720052 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:50.720265 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:50.722005 (+ 1740us) server_negotiation.cc:299] Negotiation successful
0929 20:32:50.722206 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12357,"thread_start_us":12194,"threads_started":1}
I20260929 20:32:50.725026 13003 tablet_service.cc:1467] Processing CreateTablet for tablet 9568a87b647d4fab9b44a14a4f1e53a0 (DEFAULT_TABLE table=test-table [id=3fadd99caa744faba4add65a20b24e32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:32:50.726848 13003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9568a87b647d4fab9b44a14a4f1e53a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:32:50.726904 12867 tablet_service.cc:1467] Processing CreateTablet for tablet 9568a87b647d4fab9b44a14a4f1e53a0 (DEFAULT_TABLE table=test-table [id=3fadd99caa744faba4add65a20b24e32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:32:50.727221 13140 tablet_service.cc:1467] Processing CreateTablet for tablet 9568a87b647d4fab9b44a14a4f1e53a0 (DEFAULT_TABLE table=test-table [id=3fadd99caa744faba4add65a20b24e32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:32:50.728559 13140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9568a87b647d4fab9b44a14a4f1e53a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:32:50.728595 12867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9568a87b647d4fab9b44a14a4f1e53a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:32:50.742951 13225 tablet_bootstrap.cc:492] T 9568a87b647d4fab9b44a14a4f1e53a0 P 6a86a4b292a840b58fc773f99295c67c: Bootstrap starting.
I20260929 20:32:50.746809 13225 tablet_bootstrap.cc:654] T 9568a87b647d4fab9b44a14a4f1e53a0 P 6a86a4b292a840b58fc773f99295c67c: Neither blocks nor log segments found. Creating new log.
I20260929 20:32:50.747879 13227 tablet_bootstrap.cc:492] T 9568a87b647d4fab9b44a14a4f1e53a0 P 64602944447a488d8a0ac58f14700eb6: Bootstrap starting.
I20260929 20:32:50.748361 13225 log.cc:824] T 9568a87b647d4fab9b44a14a4f1e53a0 P 6a86a4b292a840b58fc773f99295c67c: Log is configured to *not* fsync() on all Append() calls
I20260929 20:32:50.752169 13227 tablet_bootstrap.cc:654] T 9568a87b647d4fab9b44a14a4f1e53a0 P 64602944447a488d8a0ac58f14700eb6: Neither blocks nor log segments found. Creating new log.
I20260929 20:32:50.753720 13225 tablet_bootstrap.cc:492] T 9568a87b647d4fab9b44a14a4f1e53a0 P 6a86a4b292a840b58fc773f99295c67c: No bootstrap required, opened a new log
I20260929 20:32:50.753648 13227 log.cc:824] T 9568a87b647d4fab9b44a14a4f1e53a0 P 64602944447a488d8a0ac58f14700eb6: Log is configured to *not* fsync() on all Append() calls
I20260929 20:32:50.754033 13225 ts_tablet_manager.cc:1397] T 9568a87b647d4fab9b44a14a4f1e53a0 P 6a86a4b292a840b58fc773f99295c67c: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260929 20:32:50.755242 13226 tablet_bootstrap.cc:492] T 9568a87b647d4fab9b44a14a4f1e53a0 P e2ed5f2c06fb4efdb4f3e9de12ed66fb: Bootstrap starting.
I20260929 20:32:50.756979 13227 tablet_bootstrap.cc:492] T 9568a87b647d4fab9b44a14a4f1e53a0 P 64602944447a488d8a0ac58f14700eb6: No bootstrap required, opened a new log
I20260929 20:32:50.757295 13227 ts_tablet_manager.cc:1397] T 9568a87b647d4fab9b44a14a4f1e53a0 P 64602944447a488d8a0ac58f14700eb6: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260929 20:32:50.759269 13226 tablet_bootstrap.cc:654] T 9568a87b647d4fab9b44a14a4f1e53a0 P e2ed5f2c06fb4efdb4f3e9de12ed66fb: Neither blocks nor log segments found. Creating new log.
I20260929 20:32:50.760725 13226 log.cc:824] T 9568a87b647d4fab9b44a14a4f1e53a0 P e2ed5f2c06fb4efdb4f3e9de12ed66fb: Log is configured to *not* fsync() on all Append() calls
I20260929 20:32:50.764860 13226 tablet_bootstrap.cc:492] T 9568a87b647d4fab9b44a14a4f1e53a0 P e2ed5f2c06fb4efdb4f3e9de12ed66fb: No bootstrap required, opened a new log
I20260929 20:32:50.765210 13226 ts_tablet_manager.cc:1397] T 9568a87b647d4fab9b44a14a4f1e53a0 P e2ed5f2c06fb4efdb4f3e9de12ed66fb: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260929 20:32:50.765789 13227 raft_consensus.cc:348] T 9568a87b647d4fab9b44a14a4f1e53a0 P 64602944447a488d8a0ac58f14700eb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a86a4b292a840b58fc773f99295c67c" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36139 } } peers { permanent_uuid: "e2ed5f2c06fb4efdb4f3e9de12ed66fb" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33195 } } peers { permanent_uuid: "64602944447a488d8a0ac58f14700eb6" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 42005 } }
I20260929 20:32:50.765818 13225 raft_consensus.cc:348] T 9568a87b647d4fab9b44a14a4f1e53a0 P 6a86a4b292a840b58fc773f99295c67c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a86a4b292a840b58fc773f99295c67c" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36139 } } peers { permanent_uuid: "e2ed5f2c06fb4efdb4f3e9de12ed66fb" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33195 } } peers { permanent_uuid: "64602944447a488d8a0ac58f14700eb6" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 42005 } }
I20260929 20:32:50.766220 13227 raft_consensus.cc:374] T 9568a87b647d4fab9b44a14a4f1e53a0 P 64602944447a488d8a0ac58f14700eb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:32:50.766297 13225 raft_consensus.cc:374] T 9568a87b647d4fab9b44a14a4f1e53a0 P 6a86a4b292a840b58fc773f99295c67c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:32:50.766331 13227 raft_consensus.cc:729] T 9568a87b647d4fab9b44a14a4f1e53a0 P 64602944447a488d8a0ac58f14700eb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64602944447a488d8a0ac58f14700eb6, State: Initialized, Role: FOLLOWER
I20260929 20:32:50.766402 13225 raft_consensus.cc:729] T 9568a87b647d4fab9b44a14a4f1e53a0 P 6a86a4b292a840b58fc773f99295c67c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a86a4b292a840b58fc773f99295c67c, State: Initialized, Role: FOLLOWER
I20260929 20:32:50.767127 13225 consensus_queue.cc:260] T 9568a87b647d4fab9b44a14a4f1e53a0 P 6a86a4b292a840b58fc773f99295c67c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a86a4b292a840b58fc773f99295c67c" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36139 } } peers { permanent_uuid: "e2ed5f2c06fb4efdb4f3e9de12ed66fb" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33195 } } peers { permanent_uuid: "64602944447a488d8a0ac58f14700eb6" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 42005 } }
I20260929 20:32:50.767160 13227 consensus_queue.cc:260] T 9568a87b647d4fab9b44a14a4f1e53a0 P 64602944447a488d8a0ac58f14700eb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a86a4b292a840b58fc773f99295c67c" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36139 } } peers { permanent_uuid: "e2ed5f2c06fb4efdb4f3e9de12ed66fb" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33195 } } peers { permanent_uuid: "64602944447a488d8a0ac58f14700eb6" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 42005 } }
I20260929 20:32:50.776356 13225 ts_tablet_manager.cc:1428] T 9568a87b647d4fab9b44a14a4f1e53a0 P 6a86a4b292a840b58fc773f99295c67c: Time spent starting tablet: real 0.022s user 0.011s sys 0.011s
I20260929 20:32:50.777072 13205 heartbeater.cc:503] Master 127.28.244.190:35929 was elected leader, sending a full tablet report...
I20260929 20:32:50.778595 13227 ts_tablet_manager.cc:1428] T 9568a87b647d4fab9b44a14a4f1e53a0 P 64602944447a488d8a0ac58f14700eb6: Time spent starting tablet: real 0.021s user 0.018s sys 0.004s
I20260929 20:32:50.779402 13226 raft_consensus.cc:348] T 9568a87b647d4fab9b44a14a4f1e53a0 P e2ed5f2c06fb4efdb4f3e9de12ed66fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a86a4b292a840b58fc773f99295c67c" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36139 } } peers { permanent_uuid: "e2ed5f2c06fb4efdb4f3e9de12ed66fb" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33195 } } peers { permanent_uuid: "64602944447a488d8a0ac58f14700eb6" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 42005 } }
I20260929 20:32:50.779980 13226 raft_consensus.cc:374] T 9568a87b647d4fab9b44a14a4f1e53a0 P e2ed5f2c06fb4efdb4f3e9de12ed66fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:32:50.780097 13226 raft_consensus.cc:729] T 9568a87b647d4fab9b44a14a4f1e53a0 P e2ed5f2c06fb4efdb4f3e9de12ed66fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2ed5f2c06fb4efdb4f3e9de12ed66fb, State: Initialized, Role: FOLLOWER
I20260929 20:32:50.780771 13226 consensus_queue.cc:260] T 9568a87b647d4fab9b44a14a4f1e53a0 P e2ed5f2c06fb4efdb4f3e9de12ed66fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a86a4b292a840b58fc773f99295c67c" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36139 } } peers { permanent_uuid: "e2ed5f2c06fb4efdb4f3e9de12ed66fb" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33195 } } peers { permanent_uuid: "64602944447a488d8a0ac58f14700eb6" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 42005 } }
I20260929 20:32:50.783623 13226 ts_tablet_manager.cc:1428] T 9568a87b647d4fab9b44a14a4f1e53a0 P e2ed5f2c06fb4efdb4f3e9de12ed66fb: Time spent starting tablet: real 0.018s user 0.017s sys 0.000s
I20260929 20:32:50.784160 13070 heartbeater.cc:503] Master 127.28.244.190:35929 was elected leader, sending a full tablet report...
W20260929 20:32:50.793509 13071 tablet.cc:2369] T 9568a87b647d4fab9b44a14a4f1e53a0 P e2ed5f2c06fb4efdb4f3e9de12ed66fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:32:50.841176 13234 raft_consensus.cc:482] T 9568a87b647d4fab9b44a14a4f1e53a0 P e2ed5f2c06fb4efdb4f3e9de12ed66fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:32:50.841533 13234 raft_consensus.cc:504] T 9568a87b647d4fab9b44a14a4f1e53a0 P e2ed5f2c06fb4efdb4f3e9de12ed66fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a86a4b292a840b58fc773f99295c67c" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36139 } } peers { permanent_uuid: "e2ed5f2c06fb4efdb4f3e9de12ed66fb" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33195 } } peers { permanent_uuid: "64602944447a488d8a0ac58f14700eb6" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 42005 } }
I20260929 20:32:50.843468 13234 leader_election.cc:288] T 9568a87b647d4fab9b44a14a4f1e53a0 P e2ed5f2c06fb4efdb4f3e9de12ed66fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a86a4b292a840b58fc773f99295c67c (127.28.244.131:36139), 64602944447a488d8a0ac58f14700eb6 (127.28.244.129:42005)
I20260929 20:32:50.851513 13224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:50.844126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:48155 (local address 127.28.244.129:42005)
0929 20:32:50.844384 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:50.844392 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:50.845408 (+ 1016us) server_negotiation.cc:407] Connection header received
0929 20:32:50.845637 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:50.845641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:50.845706 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:50.845826 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:32:50.846705 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:50.847429 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:50.849617 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:50.850434 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:50.850607 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:50.850708 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:50.851079 (+ 371us) server_negotiation.cc:299] Negotiation successful
0929 20:32:50.851230 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 20:32:50.851538 13222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:50.843919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:40039 (local address 127.28.244.131:36139)
0929 20:32:50.844143 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:50.844149 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:50.845327 (+ 1178us) server_negotiation.cc:407] Connection header received
0929 20:32:50.845594 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:50.845602 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:50.845686 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:50.845758 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:32:50.846612 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:50.847421 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:50.849618 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:50.850471 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:50.850676 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:50.850766 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:50.851130 (+ 364us) server_negotiation.cc:299] Negotiation successful
0929 20:32:50.851265 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 20:32:50.851831 13236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:50.843793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:36139 (local address 127.28.244.130:40039)
0929 20:32:50.845077 (+ 1284us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:50.845096 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:50.845418 (+ 322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:50.845817 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:50.845822 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:50.845847 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:32:50.846427 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:50.846440 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:50.847565 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:50.847573 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:50.849431 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:50.849445 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:50.850751 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:50.850763 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:50.850936 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:50.850960 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 20:32:50.851073 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 20:32:50.851241 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1145,"mutex_wait_us":173,"thread_start_us":930,"threads_started":1}
I20260929 20:32:50.852308 13235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:50.844117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:42005 (local address 127.28.244.130:48155)
0929 20:32:50.845237 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0929 20:32:50.845253 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:32:50.845473 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:32:50.845864 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:32:50.845868 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:32:50.845893 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:32:50.846496 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:50.846542 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:50.847611 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:50.847615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:50.849422 (+ 1807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:32:50.849438 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:32:50.850705 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:32:50.850713 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:32:50.850910 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:50.850936 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 20:32:50.851017 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 20:32:50.851209 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":939,"mutex_wait_us":53,"spinlock_wait_cycles":24832,"thread_start_us":444,"threads_started":1}
I20260929 20:32:50.853083 13160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9568a87b647d4fab9b44a14a4f1e53a0" candidate_uuid: "e2ed5f2c06fb4efdb4f3e9de12ed66fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a86a4b292a840b58fc773f99295c67c" is_pre_election: true
I20260929 20:32:50.853483 12887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9568a87b647d4fab9b44a14a4f1e53a0" candidate_uuid: "e2ed5f2c06fb4efdb4f3e9de12ed66fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64602944447a488d8a0ac58f14700eb6" is_pre_election: true
I20260929 20:32:50.853776 13160 raft_consensus.cc:2454] T 9568a87b647d4fab9b44a14a4f1e53a0 P 6a86a4b292a840b58fc773f99295c67c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2ed5f2c06fb4efdb4f3e9de12ed66fb in term 0.
I20260929 20:32:50.854060 12887 raft_consensus.cc:2454] T 9568a87b647d4fab9b44a14a4f1e53a0 P 64602944447a488d8a0ac58f14700eb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2ed5f2c06fb4efdb4f3e9de12ed66fb in term 0.
I20260929 20:32:50.854609 12958 leader_election.cc:302] T 9568a87b647d4fab9b44a14a4f1e53a0 P e2ed5f2c06fb4efdb4f3e9de12ed66fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a86a4b292a840b58fc773f99295c67c, e2ed5f2c06fb4efdb4f3e9de12ed66fb; no voters:
I20260929 20:32:50.855038 13234 raft_consensus.cc:2781] T 9568a87b647d4fab9b44a14a4f1e53a0 P e2ed5f2c06fb4efdb4f3e9de12ed66fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:32:50.855193 13234 raft_consensus.cc:482] T 9568a87b647d4fab9b44a14a4f1e53a0 P e2ed5f2c06fb4efdb4f3e9de12ed66fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:32:50.855315 13234 raft_consensus.cc:3037] T 9568a87b647d4fab9b44a14a4f1e53a0 P e2ed5f2c06fb4efdb4f3e9de12ed66fb [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:32:50.857663 13234 raft_consensus.cc:504] T 9568a87b647d4fab9b44a14a4f1e53a0 P e2ed5f2c06fb4efdb4f3e9de12ed66fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a86a4b292a840b58fc773f99295c67c" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36139 } } peers { permanent_uuid: "e2ed5f2c06fb4efdb4f3e9de12ed66fb" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33195 } } peers { permanent_uuid: "64602944447a488d8a0ac58f14700eb6" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 42005 } }
I20260929 20:32:50.858378 13234 leader_election.cc:288] T 9568a87b647d4fab9b44a14a4f1e53a0 P e2ed5f2c06fb4efdb4f3e9de12ed66fb [CANDIDATE]: Term 1 election: Requested vote from peers 6a86a4b292a840b58fc773f99295c67c (127.28.244.131:36139), 64602944447a488d8a0ac58f14700eb6 (127.28.244.129:42005)
I20260929 20:32:50.858927 13160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9568a87b647d4fab9b44a14a4f1e53a0" candidate_uuid: "e2ed5f2c06fb4efdb4f3e9de12ed66fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a86a4b292a840b58fc773f99295c67c"
I20260929 20:32:50.858976 12887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9568a87b647d4fab9b44a14a4f1e53a0" candidate_uuid: "e2ed5f2c06fb4efdb4f3e9de12ed66fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64602944447a488d8a0ac58f14700eb6"
I20260929 20:32:50.859181 13160 raft_consensus.cc:3037] T 9568a87b647d4fab9b44a14a4f1e53a0 P 6a86a4b292a840b58fc773f99295c67c [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:32:50.859184 12887 raft_consensus.cc:3037] T 9568a87b647d4fab9b44a14a4f1e53a0 P 64602944447a488d8a0ac58f14700eb6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:32:50.862600 13160 raft_consensus.cc:2454] T 9568a87b647d4fab9b44a14a4f1e53a0 P 6a86a4b292a840b58fc773f99295c67c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2ed5f2c06fb4efdb4f3e9de12ed66fb in term 1.
I20260929 20:32:50.862600 12887 raft_consensus.cc:2454] T 9568a87b647d4fab9b44a14a4f1e53a0 P 64602944447a488d8a0ac58f14700eb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2ed5f2c06fb4efdb4f3e9de12ed66fb in term 1.
I20260929 20:32:50.863200 12957 leader_election.cc:302] T 9568a87b647d4fab9b44a14a4f1e53a0 P e2ed5f2c06fb4efdb4f3e9de12ed66fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 64602944447a488d8a0ac58f14700eb6, 6a86a4b292a840b58fc773f99295c67c, e2ed5f2c06fb4efdb4f3e9de12ed66fb; no voters:
I20260929 20:32:50.863607 13234 raft_consensus.cc:2781] T 9568a87b647d4fab9b44a14a4f1e53a0 P e2ed5f2c06fb4efdb4f3e9de12ed66fb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:32:50.864179 13234 raft_consensus.cc:686] T 9568a87b647d4fab9b44a14a4f1e53a0 P e2ed5f2c06fb4efdb4f3e9de12ed66fb [term 1 LEADER]: Becoming Leader. State: Replica: e2ed5f2c06fb4efdb4f3e9de12ed66fb, State: Running, Role: LEADER
W20260929 20:32:50.864845 13206 tablet.cc:2369] T 9568a87b647d4fab9b44a14a4f1e53a0 P 6a86a4b292a840b58fc773f99295c67c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:32:50.864775 13234 consensus_queue.cc:237] T 9568a87b647d4fab9b44a14a4f1e53a0 P e2ed5f2c06fb4efdb4f3e9de12ed66fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a86a4b292a840b58fc773f99295c67c" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36139 } } peers { permanent_uuid: "e2ed5f2c06fb4efdb4f3e9de12ed66fb" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33195 } } peers { permanent_uuid: "64602944447a488d8a0ac58f14700eb6" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 42005 } }
I20260929 20:32:50.871071 12742 catalog_manager.cc:5495] T 9568a87b647d4fab9b44a14a4f1e53a0 P e2ed5f2c06fb4efdb4f3e9de12ed66fb reported cstate change: term changed from 0 to 1, leader changed from <none> to e2ed5f2c06fb4efdb4f3e9de12ed66fb (127.28.244.130). New cstate: current_term: 1 leader_uuid: "e2ed5f2c06fb4efdb4f3e9de12ed66fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a86a4b292a840b58fc773f99295c67c" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2ed5f2c06fb4efdb4f3e9de12ed66fb" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64602944447a488d8a0ac58f14700eb6" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 42005 } health_report { overall_health: UNKNOWN } } }
I20260929 20:32:50.919560 13221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:50.912358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52020 (local address 127.28.244.130:33195)
0929 20:32:50.912642 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:50.912647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:50.912710 (+ 63us) server_negotiation.cc:407] Connection header received
0929 20:32:50.912975 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:50.912979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:50.913068 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:50.913229 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:32:50.913716 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:50.914703 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:50.916528 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:50.917106 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:50.917234 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:50.917956 (+ 722us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:32:50.918694 (+ 738us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:32:50.918854 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:50.919113 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 20:32:50.919210 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 20:32:50.929529 12887 raft_consensus.cc:1261] T 9568a87b647d4fab9b44a14a4f1e53a0 P 64602944447a488d8a0ac58f14700eb6 [term 1 FOLLOWER]: Refusing update from remote peer e2ed5f2c06fb4efdb4f3e9de12ed66fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:32:50.929531 13160 raft_consensus.cc:1261] T 9568a87b647d4fab9b44a14a4f1e53a0 P 6a86a4b292a840b58fc773f99295c67c [term 1 FOLLOWER]: Refusing update from remote peer e2ed5f2c06fb4efdb4f3e9de12ed66fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:32:50.930926 13237 consensus_queue.cc:1035] T 9568a87b647d4fab9b44a14a4f1e53a0 P e2ed5f2c06fb4efdb4f3e9de12ed66fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a86a4b292a840b58fc773f99295c67c" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:32:50.931372 13234 consensus_queue.cc:1035] T 9568a87b647d4fab9b44a14a4f1e53a0 P e2ed5f2c06fb4efdb4f3e9de12ed66fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "64602944447a488d8a0ac58f14700eb6" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 42005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:32:50.962714 12741 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:32:50.963068 12741 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:32:50.967357 12741 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457: Sending DeleteTablet for 3 replicas of tablet 9568a87b647d4fab9b44a14a4f1e53a0
I20260929 20:32:50.968971 13003 tablet_service.cc:1514] Processing DeleteTablet for tablet 9568a87b647d4fab9b44a14a4f1e53a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:32:50 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:52014
I20260929 20:32:50.968971 13140 tablet_service.cc:1514] Processing DeleteTablet for tablet 9568a87b647d4fab9b44a14a4f1e53a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:32:50 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:46168
I20260929 20:32:50.969853 12867 tablet_service.cc:1514] Processing DeleteTablet for tablet 9568a87b647d4fab9b44a14a4f1e53a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:32:50 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:60878
I20260929 20:32:50.970124 13251 tablet_replica.cc:330] T 9568a87b647d4fab9b44a14a4f1e53a0 P 6a86a4b292a840b58fc773f99295c67c: stopping tablet replica
I20260929 20:32:50.970696 13251 raft_consensus.cc:2229] T 9568a87b647d4fab9b44a14a4f1e53a0 P 6a86a4b292a840b58fc773f99295c67c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:32:50.970894 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 12803
I20260929 20:32:50.971468 13251 raft_consensus.cc:2258] T 9568a87b647d4fab9b44a14a4f1e53a0 P 6a86a4b292a840b58fc773f99295c67c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:32:50.974536 13251 ts_tablet_manager.cc:1905] T 9568a87b647d4fab9b44a14a4f1e53a0 P 6a86a4b292a840b58fc773f99295c67c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:32:50.984807 13251 ts_tablet_manager.cc:1918] T 9568a87b647d4fab9b44a14a4f1e53a0 P 6a86a4b292a840b58fc773f99295c67c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:32:50.985190 13251 log.cc:1196] T 9568a87b647d4fab9b44a14a4f1e53a0 P 6a86a4b292a840b58fc773f99295c67c: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/wal/wals/9568a87b647d4fab9b44a14a4f1e53a0
I20260929 20:32:50.985970 13251 ts_tablet_manager.cc:1939] T 9568a87b647d4fab9b44a14a4f1e53a0 P 6a86a4b292a840b58fc773f99295c67c: Deleting consensus metadata
I20260929 20:32:50.988713 12728 catalog_manager.cc:4841] TS 6a86a4b292a840b58fc773f99295c67c (127.28.244.131:36139): tablet 9568a87b647d4fab9b44a14a4f1e53a0 (table test-table [id=3fadd99caa744faba4add65a20b24e32]) successfully deleted
I20260929 20:32:50.996616 13253 tablet_replica.cc:330] T 9568a87b647d4fab9b44a14a4f1e53a0 P 64602944447a488d8a0ac58f14700eb6: stopping tablet replica
I20260929 20:32:50.997409 13253 raft_consensus.cc:2229] T 9568a87b647d4fab9b44a14a4f1e53a0 P 64602944447a488d8a0ac58f14700eb6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:32:50.998198 13253 raft_consensus.cc:2258] T 9568a87b647d4fab9b44a14a4f1e53a0 P 64602944447a488d8a0ac58f14700eb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:32:50.999354 13252 tablet_replica.cc:330] T 9568a87b647d4fab9b44a14a4f1e53a0 P e2ed5f2c06fb4efdb4f3e9de12ed66fb: stopping tablet replica
I20260929 20:32:51.000133 13252 raft_consensus.cc:2229] T 9568a87b647d4fab9b44a14a4f1e53a0 P e2ed5f2c06fb4efdb4f3e9de12ed66fb [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:32:51.000945 13252 raft_consensus.cc:2258] T 9568a87b647d4fab9b44a14a4f1e53a0 P e2ed5f2c06fb4efdb4f3e9de12ed66fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:32:51.001104 13253 ts_tablet_manager.cc:1905] T 9568a87b647d4fab9b44a14a4f1e53a0 P 64602944447a488d8a0ac58f14700eb6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:32:51.003554 13252 ts_tablet_manager.cc:1905] T 9568a87b647d4fab9b44a14a4f1e53a0 P e2ed5f2c06fb4efdb4f3e9de12ed66fb: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 20:32:51 dist-test-slave-p29l krb5kdc[12696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.129@KRBTEST.COM
I20260929 20:32:51.009891 13253 ts_tablet_manager.cc:1918] T 9568a87b647d4fab9b44a14a4f1e53a0 P 64602944447a488d8a0ac58f14700eb6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:32:51.010116 13253 log.cc:1196] T 9568a87b647d4fab9b44a14a4f1e53a0 P 64602944447a488d8a0ac58f14700eb6: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/wal/wals/9568a87b647d4fab9b44a14a4f1e53a0
I20260929 20:32:51.011092 13253 ts_tablet_manager.cc:1939] T 9568a87b647d4fab9b44a14a4f1e53a0 P 64602944447a488d8a0ac58f14700eb6: Deleting consensus metadata
I20260929 20:32:51.012485 13252 ts_tablet_manager.cc:1918] T 9568a87b647d4fab9b44a14a4f1e53a0 P e2ed5f2c06fb4efdb4f3e9de12ed66fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:32:51.012790 13252 log.cc:1196] T 9568a87b647d4fab9b44a14a4f1e53a0 P e2ed5f2c06fb4efdb4f3e9de12ed66fb: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/wal/wals/9568a87b647d4fab9b44a14a4f1e53a0
I20260929 20:32:51.013788 13252 ts_tablet_manager.cc:1939] T 9568a87b647d4fab9b44a14a4f1e53a0 P e2ed5f2c06fb4efdb4f3e9de12ed66fb: Deleting consensus metadata
I20260929 20:32:51.013792 12727 catalog_manager.cc:4841] TS 64602944447a488d8a0ac58f14700eb6 (127.28.244.129:42005): tablet 9568a87b647d4fab9b44a14a4f1e53a0 (table test-table [id=3fadd99caa744faba4add65a20b24e32]) successfully deleted
I20260929 20:32:51.015659 12729 catalog_manager.cc:4841] TS e2ed5f2c06fb4efdb4f3e9de12ed66fb (127.28.244.130:33195): tablet 9568a87b647d4fab9b44a14a4f1e53a0 (table test-table [id=3fadd99caa744faba4add65a20b24e32]) successfully deleted
I20260929 20:32:51.017973 13224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:50.972240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60882 (local address 127.28.244.129:42005)
0929 20:32:50.972424 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:50.972434 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:50.995782 (+ 23348us) server_negotiation.cc:407] Connection header received
0929 20:32:50.996097 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:50.996105 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:50.996246 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:50.996346 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:50.999099 (+ 2753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:51.000322 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:51.002565 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:51.003277 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:51.003387 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:51.009672 (+ 6285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:51.009694 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:51.009718 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:51.009776 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:51.014223 (+ 4447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:51.015354 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:51.015363 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:51.015369 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:51.015562 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:51.016095 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:51.016099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:51.016101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:51.016495 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:51.016753 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:51.017361 (+ 608us) server_negotiation.cc:299] Negotiation successful
0929 20:32:51.017572 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 20:32:51.018702 12928 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:32:51.177019 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 12803
I20260929 20:32:51.196300 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 12940
Sep 29 20:32:51 dist-test-slave-p29l krb5kdc[12696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.130@KRBTEST.COM
I20260929 20:32:51.218261 13221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:51.197370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52022 (local address 127.28.244.130:33195)
0929 20:32:51.197555 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:51.197559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:51.197577 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:32:51.197637 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:51.197640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:51.197726 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:51.197813 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:51.200009 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:51.200962 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:51.203213 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:51.204253 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:51.204409 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:51.210647 (+ 6238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:51.210666 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:51.210685 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:51.210740 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:51.214466 (+ 3726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:51.215469 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:51.215480 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:51.215482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:51.215666 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:51.216193 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:51.216201 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:51.216204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:51.216679 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:51.217194 (+ 515us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:51.217529 (+ 335us) server_negotiation.cc:299] Negotiation successful
0929 20:32:51.217808 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 20:32:51.219123 13065 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:32:51.343078 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 12940
I20260929 20:32:51.365514 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 13076
Sep 29 20:32:51 dist-test-slave-p29l krb5kdc[12696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.131@KRBTEST.COM
I20260929 20:32:51.383294 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:32:51.366380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46184 (local address 127.28.244.131:36139)
0929 20:32:51.366852 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0929 20:32:51.366857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:32:51.366876 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:32:51.366940 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:32:51.366944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:32:51.367012 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:32:51.367111 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:32:51.368983 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:51.369765 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:51.371306 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:32:51.371875 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:32:51.372016 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:32:51.376914 (+ 4898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:32:51.376936 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:32:51.376963 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:32:51.377010 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:32:51.380067 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:51.380982 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:51.380993 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:51.380996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:51.381163 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:32:51.381591 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:32:51.381595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:32:51.381597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:32:51.382031 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:32:51.382282 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0929 20:32:51.382732 (+ 450us) server_negotiation.cc:299] Negotiation successful
0929 20:32:51.382998 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":230,"threads_started":1}
I20260929 20:32:51.384488 13200 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:32:51.479259 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 13076
I20260929 20:32:51.497797 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 12709
I20260929 20:32:51.499100 12772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:32:51.617358 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 12709
I20260929 20:33:11.642894 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:35929
--webserver_interface=127.28.244.190
--webserver_port=34105
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:35929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:33:12.005184 13260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:33:12.005512 13260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:33:12.005698 13260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:33:12.013602 13260 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 20:33:12.013736 13260 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 20:33:12.013810 13260 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 20:33:12.013911 13260 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:33:12.014019 13260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:33:12.014115 13260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:33:12.014204 13260 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:33:12.014303 13260 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:33:12.025063 13260 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:35929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:35929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=34105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:33:12.026520 13260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:33:12.028071 13260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:33:12.053280 13266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:33:12.053184 13267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:33:12.053452 13260 server_base.cc:1034] running on GCE node
W20260929 20:33:12.053270 13269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:33:12.054757 13260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:33:12.056670 13260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:33:12.057981 13260 hybrid_clock.cc:648] HybridClock initialized: now 1790713992057915 us; error 39 us; skew 500 ppm
I20260929 20:33:12.058478 13260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:33:12 dist-test-slave-p29l krb5kdc[12696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713992, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:33:12.068286 13260 init.cc:377] Logged in from keytab as kudu/127.28.244.190@KRBTEST.COM (short username kudu)
I20260929 20:33:12.070397 13260 webserver.cc:460] Webserver started at http://127.28.244.190:34105/ using document root <none> and password file <none>
I20260929 20:33:12.071348 13260 fs_manager.cc:359] Metadata directory not provided
I20260929 20:33:12.071516 13260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:33:12.078444 13260 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 20:33:12.082191 13276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:33:12.083920 13260 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 20:33:12.084138 13260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "391fcaa2e0d84f20b5f803c250eea457"
format_stamp: "Formatted at 2026-09-29 20:32:48 on dist-test-slave-p29l"
server_key: "4b21e2e5d948004ed4c01ce708b1f639"
server_key_iv: "3ab699e47e976be32682fc4d568d3e32"
server_key_version: "encryptionkey@0"
I20260929 20:33:12.085354 13260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:33:12.102128 13260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:33:12.107169 13260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:33:12.107530 13260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:33:12.129061 13260 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:35929
I20260929 20:33:12.129091 13328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:35929 every 8 connection(s)
I20260929 20:33:12.130901 13260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:33:12.138827 13329 sys_catalog.cc:263] Verifying existing consensus state
I20260929 20:33:12.140517 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 13260
I20260929 20:33:12.141558 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.129:42005
--local_ip_for_outbound_sockets=127.28.244.129
--tserver_master_addrs=127.28.244.190:35929
--webserver_port=43257
--webserver_interface=127.28.244.129
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:33:12.143997 13329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457: Bootstrap starting.
I20260929 20:33:12.169726 13329 log.cc:824] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457: Log is configured to *not* fsync() on all Append() calls
I20260929 20:33:12.183188 13329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:33:12.183862 13329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457: Bootstrap complete.
I20260929 20:33:12.194007 13329 raft_consensus.cc:348] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391fcaa2e0d84f20b5f803c250eea457" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 35929 } }
I20260929 20:33:12.194939 13329 raft_consensus.cc:729] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 391fcaa2e0d84f20b5f803c250eea457, State: Initialized, Role: FOLLOWER
I20260929 20:33:12.195633 13329 consensus_queue.cc:260] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391fcaa2e0d84f20b5f803c250eea457" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 35929 } }
I20260929 20:33:12.195953 13329 raft_consensus.cc:388] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:33:12.196113 13329 raft_consensus.cc:482] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:33:12.196306 13329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [term 1 FOLLOWER]: Advancing to term 2
I20260929 20:33:12.200675 13329 raft_consensus.cc:504] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391fcaa2e0d84f20b5f803c250eea457" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 35929 } }
I20260929 20:33:12.201316 13329 leader_election.cc:302] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 391fcaa2e0d84f20b5f803c250eea457; no voters:
I20260929 20:33:12.202010 13329 leader_election.cc:288] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 20:33:12.202159 13333 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 20:33:12.203497 13333 raft_consensus.cc:686] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [term 2 LEADER]: Becoming Leader. State: Replica: 391fcaa2e0d84f20b5f803c250eea457, State: Running, Role: LEADER
I20260929 20:33:12.204274 13333 consensus_queue.cc:237] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391fcaa2e0d84f20b5f803c250eea457" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 35929 } }
I20260929 20:33:12.204710 13329 sys_catalog.cc:564] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:33:12.207237 13334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "391fcaa2e0d84f20b5f803c250eea457" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391fcaa2e0d84f20b5f803c250eea457" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 35929 } } }
I20260929 20:33:12.207743 13334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [sys.catalog]: This master's current role is: LEADER
I20260929 20:33:12.208181 13335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 391fcaa2e0d84f20b5f803c250eea457. Latest consensus state: current_term: 2 leader_uuid: "391fcaa2e0d84f20b5f803c250eea457" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391fcaa2e0d84f20b5f803c250eea457" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 35929 } } }
I20260929 20:33:12.208567 13335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [sys.catalog]: This master's current role is: LEADER
I20260929 20:33:12.215833 13339 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:33:12.224509 13339 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:33:12.226364 13339 catalog_manager.cc:1259] Loaded cluster ID: bdece67bdd2a4aacbc18219848e5f87a
I20260929 20:33:12.226562 13339 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:33:12.230974 13339 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:33:12.233029 13339 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457: Loaded TSK: 0
I20260929 20:33:12.273990 13339 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457: Generated new TSK 1
I20260929 20:33:12.274411 13339 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 20:33:12.544709 13331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:33:12.545045 13331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:33:12.545219 13331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:33:12.553262 13331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:33:12.553390 13331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:33:12.553565 13331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.129
I20260929 20:33:12.564193 13331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.129:42005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.244.129
--webserver_port=43257
--tserver_master_addrs=127.28.244.190:35929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.129
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:33:12.565912 13331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:33:12.567487 13331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:33:12.585112 13356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:33:12.585507 13359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:33:12.585978 13357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:33:12.587026 13331 server_base.cc:1034] running on GCE node
I20260929 20:33:12.587795 13331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:33:12.588999 13331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:33:12.590354 13331 hybrid_clock.cc:648] HybridClock initialized: now 1790713992590281 us; error 65 us; skew 500 ppm
I20260929 20:33:12.590863 13331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:33:12 dist-test-slave-p29l krb5kdc[12696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713992, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:33:12.594842 13331 init.cc:377] Logged in from keytab as kudu/127.28.244.129@KRBTEST.COM (short username kudu)
I20260929 20:33:12.596730 13331 webserver.cc:460] Webserver started at http://127.28.244.129:43257/ using document root <none> and password file <none>
I20260929 20:33:12.597538 13331 fs_manager.cc:359] Metadata directory not provided
I20260929 20:33:12.597729 13331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:33:12.604766 13331 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260929 20:33:12.608409 13366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:33:12.610172 13331 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260929 20:33:12.610404 13331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/wal
uuid: "64602944447a488d8a0ac58f14700eb6"
format_stamp: "Formatted at 2026-09-29 20:32:49 on dist-test-slave-p29l"
server_key: "971a8af187bd30975ba0755262c6c45d"
server_key_iv: "375b1677fced77eb8228f478c9e3bf40"
server_key_version: "encryptionkey@0"
I20260929 20:33:12.611542 13331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:33:12.633536 13331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:33:12.638672 13331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:33:12.639022 13331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:33:12.640232 13331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:33:12.641794 13331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:33:12.641932 13331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:33:12.642084 13331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:33:12.642174 13331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:33:12.679184 13331 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.129:42005
I20260929 20:33:12.679248 13480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.129:42005 every 8 connection(s)
I20260929 20:33:12.680816 13331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
I20260929 20:33:12.685388 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 13331
I20260929 20:33:12.686519 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.130:33195
--local_ip_for_outbound_sockets=127.28.244.130
--tserver_master_addrs=127.28.244.190:35929
--webserver_port=42909
--webserver_interface=127.28.244.130
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:33:12 dist-test-slave-p29l krb5kdc[12696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713992, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:33:12.708280 13484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:12.684421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.129:59521)
0929 20:33:12.685293 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:12.685336 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 20:33:12.687395 (+ 2059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:33:12.689087 (+ 1692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:33:12.689100 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:33:12.689862 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:33:12.690533 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:12.690558 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:12.692528 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:12.692532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:12.693825 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:12.693840 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:12.695008 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:12.695016 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:12.695316 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:12.696134 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:33:12.696164 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:33:12.698851 (+ 2687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:33:12.702970 (+ 4119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:33:12.702981 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:33:12.703006 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:33:12.703500 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:33:12.704152 (+ 652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:33:12.704162 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:33:12.704164 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:33:12.704384 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:33:12.705412 (+ 1028us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:33:12.705421 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:33:12.705815 (+ 394us) client_negotiation.cc:769] Sending connection context
0929 20:33:12.706218 (+ 403us) client_negotiation.cc:240] Negotiation successful
0929 20:33:12.706835 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":398,"threads_started":1}
I20260929 20:33:12.708281 13483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:12.684066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:59521 (local address 127.28.244.190:35929)
0929 20:33:12.684674 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:12.684687 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:12.685487 (+ 800us) server_negotiation.cc:407] Connection header received
0929 20:33:12.688090 (+ 2603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:12.688104 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:12.688503 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:12.688923 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:33:12.690839 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:12.692309 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:12.694023 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:12.694696 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:12.694938 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:12.699220 (+ 4282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:33:12.699231 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:33:12.699249 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:33:12.699300 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:33:12.702775 (+ 3475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:12.703674 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:12.703738 (+ 64us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:12.703744 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:12.703996 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:12.704513 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:12.704519 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:12.704523 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:12.705247 (+ 724us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:33:12.705459 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:12.706400 (+ 941us) server_negotiation.cc:299] Negotiation successful
0929 20:33:12.706835 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":256,"threads_started":1}
I20260929 20:33:12.712008 13481 heartbeater.cc:347] Connected to a master server at 127.28.244.190:35929
I20260929 20:33:12.712414 13481 heartbeater.cc:464] Registering TS with master...
I20260929 20:33:12.713466 13481 heartbeater.cc:511] Master 127.28.244.190:35929 requested a full tablet report, sending...
I20260929 20:33:12.716450 13293 ts_manager.cc:194] Registered new tserver with Master: 64602944447a488d8a0ac58f14700eb6 (127.28.244.129:42005)
I20260929 20:33:12.718698 13293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.129@KRBTEST.COM'} at 127.28.244.129:59521
W20260929 20:33:13.046209 13485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:33:13.046527 13485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:33:13.046626 13485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:33:13.054489 13485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:33:13.054634 13485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:33:13.054759 13485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.130
I20260929 20:33:13.065032 13485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.130:33195
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.244.130
--webserver_port=42909
--tserver_master_addrs=127.28.244.190:35929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.130
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:33:13.066416 13485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:33:13.067775 13485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:33:13.082579 13491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:33:13.082579 13490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:33:13.082608 13493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:33:13.083935 13485 server_base.cc:1034] running on GCE node
I20260929 20:33:13.084547 13485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:33:13.085567 13485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:33:13.086826 13485 hybrid_clock.cc:648] HybridClock initialized: now 1790713993086753 us; error 60 us; skew 500 ppm
I20260929 20:33:13.087208 13485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:33:13 dist-test-slave-p29l krb5kdc[12696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713993, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:33:13.090888 13485 init.cc:377] Logged in from keytab as kudu/127.28.244.130@KRBTEST.COM (short username kudu)
I20260929 20:33:13.092628 13485 webserver.cc:460] Webserver started at http://127.28.244.130:42909/ using document root <none> and password file <none>
I20260929 20:33:13.093501 13485 fs_manager.cc:359] Metadata directory not provided
I20260929 20:33:13.093633 13485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:33:13.099951 13485 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 20:33:13.103597 13500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:33:13.105513 13485 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 20:33:13.105754 13485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/wal
uuid: "e2ed5f2c06fb4efdb4f3e9de12ed66fb"
format_stamp: "Formatted at 2026-09-29 20:32:49 on dist-test-slave-p29l"
server_key: "aa0c4959742f053c7875eb4cebcb0a40"
server_key_iv: "f208ac299da3e23631a51bbe7394b0f4"
server_key_version: "encryptionkey@0"
I20260929 20:33:13.106999 13485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:33:13.123509 13485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:33:13.128612 13485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:33:13.128921 13485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:33:13.130069 13485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:33:13.131577 13485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:33:13.131750 13485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:33:13.131918 13485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:33:13.132010 13485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:33:13.168370 13485 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.130:33195
I20260929 20:33:13.168463 13614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.130:33195 every 8 connection(s)
I20260929 20:33:13.170011 13485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
I20260929 20:33:13.175158 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 13485
I20260929 20:33:13.175932 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.131:36139
--local_ip_for_outbound_sockets=127.28.244.131
--tserver_master_addrs=127.28.244.190:35929
--webserver_port=45243
--webserver_interface=127.28.244.131
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:33:13 dist-test-slave-p29l krb5kdc[12696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713993, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:33:13.194866 13483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:13.173131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:53173 (local address 127.28.244.190:35929)
0929 20:33:13.173320 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:13.173325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:13.174617 (+ 1292us) server_negotiation.cc:407] Connection header received
0929 20:33:13.176839 (+ 2222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:13.176845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:13.176972 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:13.177098 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:33:13.178798 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:13.179541 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:13.181671 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:13.182121 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:13.182284 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:13.187536 (+ 5252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:33:13.187542 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:33:13.187545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:33:13.187587 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:33:13.190602 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:13.191877 (+ 1275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:13.191882 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:13.191885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:13.192035 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:13.192651 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:13.192655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:13.192657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:13.193004 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:33:13.193125 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:13.194261 (+ 1136us) server_negotiation.cc:299] Negotiation successful
0929 20:33:13.194553 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:33:13.196530 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:13.173648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.130:53173)
0929 20:33:13.174415 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:13.174476 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 20:33:13.176467 (+ 1991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:33:13.177369 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:33:13.177378 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:33:13.177965 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:33:13.178592 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:13.178610 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:13.179734 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:13.179739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:13.181458 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:13.181476 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:13.182398 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:13.182405 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:13.182667 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:13.183468 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:33:13.183496 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:33:13.187024 (+ 3528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:33:13.190805 (+ 3781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:33:13.190819 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:33:13.190844 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:33:13.191612 (+ 768us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:33:13.192165 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:33:13.192174 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:33:13.192180 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:33:13.192503 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:33:13.193161 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:33:13.193173 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:33:13.193735 (+ 562us) client_negotiation.cc:769] Sending connection context
0929 20:33:13.194102 (+ 367us) client_negotiation.cc:240] Negotiation successful
0929 20:33:13.194773 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":305,"threads_started":1}
I20260929 20:33:13.198491 13615 heartbeater.cc:347] Connected to a master server at 127.28.244.190:35929
I20260929 20:33:13.198928 13615 heartbeater.cc:464] Registering TS with master...
I20260929 20:33:13.200091 13615 heartbeater.cc:511] Master 127.28.244.190:35929 requested a full tablet report, sending...
I20260929 20:33:13.202123 13293 ts_manager.cc:194] Registered new tserver with Master: e2ed5f2c06fb4efdb4f3e9de12ed66fb (127.28.244.130:33195)
I20260929 20:33:13.203261 13293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.130@KRBTEST.COM'} at 127.28.244.130:53173
W20260929 20:33:13.551664 13618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:33:13.552141 13618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:33:13.552341 13618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:33:13.560911 13618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:33:13.561118 13618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:33:13.561300 13618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.131
I20260929 20:33:13.573035 13618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.131:36139
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.244.131
--webserver_port=45243
--tserver_master_addrs=127.28.244.190:35929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.131
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:33:13.574596 13618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:33:13.576203 13618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:33:13.591446 13623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:33:13.591660 13624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:33:13.591926 13626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:33:13.592574 13618 server_base.cc:1034] running on GCE node
I20260929 20:33:13.593284 13618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:33:13.594342 13618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:33:13.595614 13618 hybrid_clock.cc:648] HybridClock initialized: now 1790713993595543 us; error 61 us; skew 500 ppm
I20260929 20:33:13.596088 13618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:33:13 dist-test-slave-p29l krb5kdc[12696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713993, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:33:13.599555 13618 init.cc:377] Logged in from keytab as kudu/127.28.244.131@KRBTEST.COM (short username kudu)
I20260929 20:33:13.601392 13618 webserver.cc:460] Webserver started at http://127.28.244.131:45243/ using document root <none> and password file <none>
I20260929 20:33:13.602283 13618 fs_manager.cc:359] Metadata directory not provided
I20260929 20:33:13.602423 13618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:33:13.609185 13618 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 20:33:13.612689 13633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:33:13.614409 13618 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 20:33:13.614620 13618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/wal
uuid: "6a86a4b292a840b58fc773f99295c67c"
format_stamp: "Formatted at 2026-09-29 20:32:50 on dist-test-slave-p29l"
server_key: "3324456691340329677eae6bf32be348"
server_key_iv: "79e5be5939164f1c12c25b5a58dda085"
server_key_version: "encryptionkey@0"
I20260929 20:33:13.615830 13618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:33:13.642108 13618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:33:13.647030 13618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:33:13.647353 13618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:33:13.648510 13618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:33:13.650156 13618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:33:13.650296 13618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:33:13.650429 13618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:33:13.650516 13618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:33:13.687302 13618 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.131:36139
I20260929 20:33:13.687352 13747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.131:36139 every 8 connection(s)
I20260929 20:33:13.688861 13618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
I20260929 20:33:13.696736 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 13618
Sep 29 20:33:13 dist-test-slave-p29l krb5kdc[12696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713993, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:33:13.713948 13483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:13.691951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:56209 (local address 127.28.244.190:35929)
0929 20:33:13.692172 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:13.692180 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:13.693455 (+ 1275us) server_negotiation.cc:407] Connection header received
0929 20:33:13.695428 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:13.695436 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:13.695557 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:13.695759 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:33:13.697974 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:13.698764 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:13.701038 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:13.701555 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:13.701673 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:13.706469 (+ 4796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:33:13.706474 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:33:13.706477 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:33:13.706514 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:33:13.709773 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:13.710891 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:13.710896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:13.710898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:13.711055 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:13.711596 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:13.711601 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:13.711603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:13.712081 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:33:13.712256 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:13.713301 (+ 1045us) server_negotiation.cc:299] Negotiation successful
0929 20:33:13.713579 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 20:33:13.715396 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:13.692370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.131:56209)
0929 20:33:13.693180 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:13.693311 (+ 131us) client_negotiation.cc:174] Beginning negotiation
0929 20:33:13.695078 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:33:13.696114 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:33:13.696142 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:33:13.696862 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:33:13.697776 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:13.697800 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:13.698960 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:13.698968 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:13.700854 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:13.700871 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:13.701835 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:13.701845 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:13.702177 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:13.703148 (+ 971us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:33:13.703175 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:33:13.706203 (+ 3028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:33:13.709995 (+ 3792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:33:13.710016 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:33:13.710044 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:33:13.710638 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:33:13.711181 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:33:13.711188 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:33:13.711192 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:33:13.711460 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:33:13.712225 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:33:13.712234 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:33:13.712796 (+ 562us) client_negotiation.cc:769] Sending connection context
0929 20:33:13.713147 (+ 351us) client_negotiation.cc:240] Negotiation successful
0929 20:33:13.713760 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":355,"threads_started":1}
I20260929 20:33:13.715523 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:13.697746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58504 (local address 127.28.244.190:35929)
0929 20:33:13.700262 (+ 2516us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:13.700267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:13.700283 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:33:13.700329 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:13.700333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:13.700389 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:13.700488 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:33:13.702586 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:13.703552 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:13.705505 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:13.706028 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:13.706135 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:13.709181 (+ 3046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:33:13.709190 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:33:13.709195 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:33:13.709268 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:33:13.712530 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:13.713457 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:13.713461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:13.713465 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:13.713601 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:13.714056 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:13.714063 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:13.714068 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:13.714417 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:33:13.714628 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:13.715048 (+ 420us) server_negotiation.cc:299] Negotiation successful
0929 20:33:13.715226 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2409,"thread_start_us":2276,"threads_started":1}
I20260929 20:33:13.717087 13749 heartbeater.cc:347] Connected to a master server at 127.28.244.190:35929
I20260929 20:33:13.717504 13749 heartbeater.cc:464] Registering TS with master...
I20260929 20:33:13.718281 13749 heartbeater.cc:511] Master 127.28.244.190:35929 requested a full tablet report, sending...
I20260929 20:33:13.720109 13292 ts_manager.cc:194] Registered new tserver with Master: 6a86a4b292a840b58fc773f99295c67c (127.28.244.131:36139)
I20260929 20:33:13.721094 13292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.131@KRBTEST.COM'} at 127.28.244.131:56209
I20260929 20:33:13.723429 13481 heartbeater.cc:503] Master 127.28.244.190:35929 was elected leader, sending a full tablet report...
I20260929 20:33:13.731534 29650 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:33:13.738976 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:13.732650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58514 (local address 127.28.244.190:35929)
0929 20:33:13.732941 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:13.732947 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:13.733287 (+ 340us) server_negotiation.cc:407] Connection header received
0929 20:33:13.733535 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:13.733539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:13.733711 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:13.733860 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:33:13.734367 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:13.735128 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:13.736871 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:13.737358 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:13.737468 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:13.738125 (+ 657us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:33:13.738387 (+ 262us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 20:33:13.738725 (+ 338us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58514: token expired
Metrics: {"server-negotiator.queue_time_us":98}
W20260929 20:33:13.739140 13752 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58514: token expired
W20260929 20:33:13.739903 13211 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.28.244.190:35929)
I20260929 20:33:13.754114 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:13.740567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58522 (local address 127.28.244.190:35929)
0929 20:33:13.740777 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:13.740782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:13.740796 (+ 14us) server_negotiation.cc:407] Connection header received
0929 20:33:13.740839 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:13.740843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:13.740904 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:13.741015 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:33:13.742525 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:13.743270 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:13.744693 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:13.745097 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:13.745268 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:13.748151 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:33:13.748157 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:33:13.748160 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:33:13.748206 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:33:13.750895 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:13.751782 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:13.751789 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:13.751792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:13.751988 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:13.752410 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:13.752415 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:13.752419 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:13.752874 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:33:13.753149 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:13.753485 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 20:33:13.753731 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 20:33:13.758425 13292 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:33:13.766237 13292 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:33:13.793267 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:13.780077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34562 (local address 127.28.244.129:42005)
0929 20:33:13.781396 (+ 1319us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:13.781406 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:13.782732 (+ 1326us) server_negotiation.cc:407] Connection header received
0929 20:33:13.783067 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:13.783077 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:13.783351 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:13.783549 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:33:13.784949 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:13.786584 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:13.790032 (+ 3448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:13.791609 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:13.791850 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:13.792027 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:13.792680 (+ 653us) server_negotiation.cc:299] Negotiation successful
0929 20:33:13.792889 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1087,"thread_start_us":1113,"threads_started":1}
I20260929 20:33:13.793502 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:13.780971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41368 (local address 127.28.244.130:33195)
0929 20:33:13.781605 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:13.781617 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:13.783507 (+ 1890us) server_negotiation.cc:407] Connection header received
0929 20:33:13.783679 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:13.783687 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:13.783962 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:13.784174 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:33:13.785260 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:13.786435 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:13.790233 (+ 3798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:13.791515 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:13.791721 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:13.791973 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:13.792944 (+ 971us) server_negotiation.cc:299] Negotiation successful
0929 20:33:13.793125 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":637,"threads_started":1}
I20260929 20:33:13.793852 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:13.780558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:36139 (local address 127.0.0.1:47974)
0929 20:33:13.782652 (+ 2094us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:13.782729 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0929 20:33:13.782875 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:33:13.783714 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:33:13.783722 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:33:13.783759 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:33:13.785275 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:13.785289 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:13.787147 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:13.787153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:13.790432 (+ 3279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:13.790448 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:13.792927 (+ 2479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:13.792933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:13.793111 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:13.793141 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 20:33:13.793232 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 20:33:13.793392 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1966,"mutex_wait_us":69,"thread_start_us":2596,"threads_started":1}
I20260929 20:33:13.794490 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:13.781295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47974 (local address 127.28.244.131:36139)
0929 20:33:13.782037 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:13.782045 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:13.783000 (+ 955us) server_negotiation.cc:407] Connection header received
0929 20:33:13.783091 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:13.783096 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:13.783351 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:13.783550 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:33:13.785500 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:13.786991 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:13.790712 (+ 3721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:13.792295 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:13.792485 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:13.792714 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:13.794089 (+ 1375us) server_negotiation.cc:299] Negotiation successful
0929 20:33:13.794251 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":287,"threads_started":1}
I20260929 20:33:13.794195 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:13.780966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:33195 (local address 127.0.0.1:41368)
0929 20:33:13.783377 (+ 2411us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:13.783399 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:33:13.783540 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:33:13.784151 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:33:13.784155 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:33:13.784175 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:33:13.784808 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:13.784823 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:13.786581 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:13.786589 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:13.790052 (+ 3463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:13.790071 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:13.791761 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:13.791768 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:13.792012 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:13.792075 (+ 63us) client_negotiation.cc:769] Sending connection context
0929 20:33:13.792233 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 20:33:13.793725 (+ 1492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2278,"mutex_wait_us":486,"thread_start_us":971,"threads_started":1}
I20260929 20:33:13.794440 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:13.779872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:42005 (local address 127.0.0.1:34562)
0929 20:33:13.782413 (+ 2541us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:13.782477 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 20:33:13.782767 (+ 290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:33:13.783888 (+ 1121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:33:13.783897 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:33:13.783923 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:33:13.784715 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:13.784734 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:13.786746 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:13.786753 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:13.789796 (+ 3043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:13.789815 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:13.792221 (+ 2406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:13.792228 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:13.792426 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:13.792470 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 20:33:13.793893 (+ 1423us) client_negotiation.cc:240] Negotiation successful
0929 20:33:13.794073 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2275,"mutex_wait_us":322,"thread_start_us":1189,"threads_started":1}
I20260929 20:33:13.798588 13683 tablet_service.cc:1467] Processing CreateTablet for tablet fb6da3a61d07493d81b877db2b8d0c5d (DEFAULT_TABLE table=test-table [id=c50add00c3c44251a7a9f2ee3107454f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:33:13.798581 13549 tablet_service.cc:1467] Processing CreateTablet for tablet fb6da3a61d07493d81b877db2b8d0c5d (DEFAULT_TABLE table=test-table [id=c50add00c3c44251a7a9f2ee3107454f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:33:13.800099 13683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb6da3a61d07493d81b877db2b8d0c5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:33:13.800122 13549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb6da3a61d07493d81b877db2b8d0c5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:33:13.803133 13415 tablet_service.cc:1467] Processing CreateTablet for tablet fb6da3a61d07493d81b877db2b8d0c5d (DEFAULT_TABLE table=test-table [id=c50add00c3c44251a7a9f2ee3107454f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:33:13.805078 13415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb6da3a61d07493d81b877db2b8d0c5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:33:13.814360 13765 tablet_bootstrap.cc:492] T fb6da3a61d07493d81b877db2b8d0c5d P 6a86a4b292a840b58fc773f99295c67c: Bootstrap starting.
I20260929 20:33:13.817167 13766 tablet_bootstrap.cc:492] T fb6da3a61d07493d81b877db2b8d0c5d P e2ed5f2c06fb4efdb4f3e9de12ed66fb: Bootstrap starting.
I20260929 20:33:13.819072 13765 tablet_bootstrap.cc:654] T fb6da3a61d07493d81b877db2b8d0c5d P 6a86a4b292a840b58fc773f99295c67c: Neither blocks nor log segments found. Creating new log.
I20260929 20:33:13.820674 13765 log.cc:824] T fb6da3a61d07493d81b877db2b8d0c5d P 6a86a4b292a840b58fc773f99295c67c: Log is configured to *not* fsync() on all Append() calls
I20260929 20:33:13.821668 13766 tablet_bootstrap.cc:654] T fb6da3a61d07493d81b877db2b8d0c5d P e2ed5f2c06fb4efdb4f3e9de12ed66fb: Neither blocks nor log segments found. Creating new log.
I20260929 20:33:13.823303 13768 tablet_bootstrap.cc:492] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6: Bootstrap starting.
I20260929 20:33:13.823493 13766 log.cc:824] T fb6da3a61d07493d81b877db2b8d0c5d P e2ed5f2c06fb4efdb4f3e9de12ed66fb: Log is configured to *not* fsync() on all Append() calls
I20260929 20:33:13.827873 13768 tablet_bootstrap.cc:654] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6: Neither blocks nor log segments found. Creating new log.
I20260929 20:33:13.828393 13765 tablet_bootstrap.cc:492] T fb6da3a61d07493d81b877db2b8d0c5d P 6a86a4b292a840b58fc773f99295c67c: No bootstrap required, opened a new log
I20260929 20:33:13.828420 13766 tablet_bootstrap.cc:492] T fb6da3a61d07493d81b877db2b8d0c5d P e2ed5f2c06fb4efdb4f3e9de12ed66fb: No bootstrap required, opened a new log
I20260929 20:33:13.828734 13765 ts_tablet_manager.cc:1397] T fb6da3a61d07493d81b877db2b8d0c5d P 6a86a4b292a840b58fc773f99295c67c: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.003s
I20260929 20:33:13.828734 13766 ts_tablet_manager.cc:1397] T fb6da3a61d07493d81b877db2b8d0c5d P e2ed5f2c06fb4efdb4f3e9de12ed66fb: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260929 20:33:13.829233 13768 log.cc:824] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6: Log is configured to *not* fsync() on all Append() calls
I20260929 20:33:13.832777 13768 tablet_bootstrap.cc:492] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6: No bootstrap required, opened a new log
I20260929 20:33:13.833141 13768 ts_tablet_manager.cc:1397] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260929 20:33:13.840528 13766 raft_consensus.cc:348] T fb6da3a61d07493d81b877db2b8d0c5d P e2ed5f2c06fb4efdb4f3e9de12ed66fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64602944447a488d8a0ac58f14700eb6" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 42005 } } peers { permanent_uuid: "6a86a4b292a840b58fc773f99295c67c" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36139 } } peers { permanent_uuid: "e2ed5f2c06fb4efdb4f3e9de12ed66fb" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33195 } }
I20260929 20:33:13.840814 13765 raft_consensus.cc:348] T fb6da3a61d07493d81b877db2b8d0c5d P 6a86a4b292a840b58fc773f99295c67c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64602944447a488d8a0ac58f14700eb6" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 42005 } } peers { permanent_uuid: "6a86a4b292a840b58fc773f99295c67c" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36139 } } peers { permanent_uuid: "e2ed5f2c06fb4efdb4f3e9de12ed66fb" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33195 } }
I20260929 20:33:13.841125 13766 raft_consensus.cc:374] T fb6da3a61d07493d81b877db2b8d0c5d P e2ed5f2c06fb4efdb4f3e9de12ed66fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:33:13.840907 13768 raft_consensus.cc:348] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64602944447a488d8a0ac58f14700eb6" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 42005 } } peers { permanent_uuid: "6a86a4b292a840b58fc773f99295c67c" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36139 } } peers { permanent_uuid: "e2ed5f2c06fb4efdb4f3e9de12ed66fb" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33195 } }
I20260929 20:33:13.841277 13765 raft_consensus.cc:374] T fb6da3a61d07493d81b877db2b8d0c5d P 6a86a4b292a840b58fc773f99295c67c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:33:13.841279 13766 raft_consensus.cc:729] T fb6da3a61d07493d81b877db2b8d0c5d P e2ed5f2c06fb4efdb4f3e9de12ed66fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2ed5f2c06fb4efdb4f3e9de12ed66fb, State: Initialized, Role: FOLLOWER
I20260929 20:33:13.841360 13768 raft_consensus.cc:374] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:33:13.841408 13765 raft_consensus.cc:729] T fb6da3a61d07493d81b877db2b8d0c5d P 6a86a4b292a840b58fc773f99295c67c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a86a4b292a840b58fc773f99295c67c, State: Initialized, Role: FOLLOWER
I20260929 20:33:13.841499 13768 raft_consensus.cc:729] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64602944447a488d8a0ac58f14700eb6, State: Initialized, Role: FOLLOWER
I20260929 20:33:13.842108 13765 consensus_queue.cc:260] T fb6da3a61d07493d81b877db2b8d0c5d P 6a86a4b292a840b58fc773f99295c67c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64602944447a488d8a0ac58f14700eb6" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 42005 } } peers { permanent_uuid: "6a86a4b292a840b58fc773f99295c67c" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36139 } } peers { permanent_uuid: "e2ed5f2c06fb4efdb4f3e9de12ed66fb" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33195 } }
I20260929 20:33:13.842031 13766 consensus_queue.cc:260] T fb6da3a61d07493d81b877db2b8d0c5d P e2ed5f2c06fb4efdb4f3e9de12ed66fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64602944447a488d8a0ac58f14700eb6" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 42005 } } peers { permanent_uuid: "6a86a4b292a840b58fc773f99295c67c" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36139 } } peers { permanent_uuid: "e2ed5f2c06fb4efdb4f3e9de12ed66fb" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33195 } }
I20260929 20:33:13.842511 13768 consensus_queue.cc:260] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64602944447a488d8a0ac58f14700eb6" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 42005 } } peers { permanent_uuid: "6a86a4b292a840b58fc773f99295c67c" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36139 } } peers { permanent_uuid: "e2ed5f2c06fb4efdb4f3e9de12ed66fb" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33195 } }
I20260929 20:33:13.843567 13749 heartbeater.cc:503] Master 127.28.244.190:35929 was elected leader, sending a full tablet report...
I20260929 20:33:13.844623 13765 ts_tablet_manager.cc:1428] T fb6da3a61d07493d81b877db2b8d0c5d P 6a86a4b292a840b58fc773f99295c67c: Time spent starting tablet: real 0.016s user 0.015s sys 0.001s
I20260929 20:33:13.846076 13615 heartbeater.cc:503] Master 127.28.244.190:35929 was elected leader, sending a full tablet report...
I20260929 20:33:13.846652 13766 ts_tablet_manager.cc:1428] T fb6da3a61d07493d81b877db2b8d0c5d P e2ed5f2c06fb4efdb4f3e9de12ed66fb: Time spent starting tablet: real 0.018s user 0.017s sys 0.000s
I20260929 20:33:13.848778 13768 ts_tablet_manager.cc:1428] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6: Time spent starting tablet: real 0.015s user 0.006s sys 0.006s
I20260929 20:33:13.849702 13773 raft_consensus.cc:482] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:33:13.850008 13773 raft_consensus.cc:504] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64602944447a488d8a0ac58f14700eb6" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 42005 } } peers { permanent_uuid: "6a86a4b292a840b58fc773f99295c67c" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36139 } } peers { permanent_uuid: "e2ed5f2c06fb4efdb4f3e9de12ed66fb" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33195 } }
I20260929 20:33:13.852403 13773 leader_election.cc:288] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a86a4b292a840b58fc773f99295c67c (127.28.244.131:36139), e2ed5f2c06fb4efdb4f3e9de12ed66fb (127.28.244.130:33195)
I20260929 20:33:13.859669 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:13.853025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:45483 (local address 127.28.244.131:36139)
0929 20:33:13.853244 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:13.853253 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:13.853739 (+ 486us) server_negotiation.cc:407] Connection header received
0929 20:33:13.854045 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:13.854052 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:13.854131 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:13.854328 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:33:13.855133 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:13.855961 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:13.857844 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:13.858670 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:13.858872 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:13.858955 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:13.859255 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 20:33:13.859389 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 20:33:13.859884 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:13.852487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:36139 (local address 127.28.244.129:45483)
0929 20:33:13.853622 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:13.853642 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:33:13.853855 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:33:13.854577 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:33:13.854581 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:33:13.854606 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:33:13.854981 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:13.854990 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:13.856099 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:13.856105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:13.857675 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:13.857688 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:13.858887 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:13.858894 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:13.859073 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:13.859099 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 20:33:13.859218 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 20:33:13.859364 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":995,"mutex_wait_us":3,"thread_start_us":730,"threads_started":1}
I20260929 20:33:13.860425 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:13.853021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:42301 (local address 127.28.244.130:33195)
0929 20:33:13.853317 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:13.853323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:13.854760 (+ 1437us) server_negotiation.cc:407] Connection header received
0929 20:33:13.854971 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:13.854977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:13.855053 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:13.855154 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:33:13.855677 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:13.856472 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:13.858411 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:13.859283 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:13.859493 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:13.859572 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:13.859971 (+ 399us) server_negotiation.cc:299] Negotiation successful
0929 20:33:13.860123 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260929 20:33:13.861228 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:13.852920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:33195 (local address 127.28.244.129:42301)
0929 20:33:13.854646 (+ 1726us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:13.854664 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:33:13.854822 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:33:13.855170 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:33:13.855174 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:33:13.855199 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:33:13.855525 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:13.855534 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:13.856615 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:13.856621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:13.858212 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:13.858226 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:13.859539 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:13.859547 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:13.859714 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:13.859738 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 20:33:13.860655 (+ 917us) client_negotiation.cc:240] Negotiation successful
0929 20:33:13.860852 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1597,"mutex_wait_us":25,"thread_start_us":1431,"threads_started":1}
I20260929 20:33:13.861303 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6da3a61d07493d81b877db2b8d0c5d" candidate_uuid: "64602944447a488d8a0ac58f14700eb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a86a4b292a840b58fc773f99295c67c" is_pre_election: true
I20260929 20:33:13.861908 13703 raft_consensus.cc:2454] T fb6da3a61d07493d81b877db2b8d0c5d P 6a86a4b292a840b58fc773f99295c67c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64602944447a488d8a0ac58f14700eb6 in term 0.
I20260929 20:33:13.862318 13569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6da3a61d07493d81b877db2b8d0c5d" candidate_uuid: "64602944447a488d8a0ac58f14700eb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2ed5f2c06fb4efdb4f3e9de12ed66fb" is_pre_election: true
I20260929 20:33:13.862864 13369 leader_election.cc:302] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64602944447a488d8a0ac58f14700eb6, 6a86a4b292a840b58fc773f99295c67c; no voters:
I20260929 20:33:13.862884 13569 raft_consensus.cc:2454] T fb6da3a61d07493d81b877db2b8d0c5d P e2ed5f2c06fb4efdb4f3e9de12ed66fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64602944447a488d8a0ac58f14700eb6 in term 0.
I20260929 20:33:13.863317 13773 raft_consensus.cc:2781] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:33:13.863495 13773 raft_consensus.cc:482] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:33:13.863623 13773 raft_consensus.cc:3037] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:33:13.866295 13773 raft_consensus.cc:504] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64602944447a488d8a0ac58f14700eb6" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 42005 } } peers { permanent_uuid: "6a86a4b292a840b58fc773f99295c67c" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36139 } } peers { permanent_uuid: "e2ed5f2c06fb4efdb4f3e9de12ed66fb" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33195 } }
I20260929 20:33:13.866918 13773 leader_election.cc:288] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [CANDIDATE]: Term 1 election: Requested vote from peers 6a86a4b292a840b58fc773f99295c67c (127.28.244.131:36139), e2ed5f2c06fb4efdb4f3e9de12ed66fb (127.28.244.130:33195)
I20260929 20:33:13.867563 13569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6da3a61d07493d81b877db2b8d0c5d" candidate_uuid: "64602944447a488d8a0ac58f14700eb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2ed5f2c06fb4efdb4f3e9de12ed66fb"
I20260929 20:33:13.867762 13569 raft_consensus.cc:3037] T fb6da3a61d07493d81b877db2b8d0c5d P e2ed5f2c06fb4efdb4f3e9de12ed66fb [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:33:13.868110 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6da3a61d07493d81b877db2b8d0c5d" candidate_uuid: "64602944447a488d8a0ac58f14700eb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a86a4b292a840b58fc773f99295c67c"
I20260929 20:33:13.868366 13703 raft_consensus.cc:3037] T fb6da3a61d07493d81b877db2b8d0c5d P 6a86a4b292a840b58fc773f99295c67c [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:33:13.871445 13569 raft_consensus.cc:2454] T fb6da3a61d07493d81b877db2b8d0c5d P e2ed5f2c06fb4efdb4f3e9de12ed66fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64602944447a488d8a0ac58f14700eb6 in term 1.
I20260929 20:33:13.871758 13703 raft_consensus.cc:2454] T fb6da3a61d07493d81b877db2b8d0c5d P 6a86a4b292a840b58fc773f99295c67c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64602944447a488d8a0ac58f14700eb6 in term 1.
I20260929 20:33:13.872114 13370 leader_election.cc:302] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64602944447a488d8a0ac58f14700eb6, e2ed5f2c06fb4efdb4f3e9de12ed66fb; no voters:
I20260929 20:33:13.872582 13773 raft_consensus.cc:2781] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:33:13.873183 13773 raft_consensus.cc:686] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [term 1 LEADER]: Becoming Leader. State: Replica: 64602944447a488d8a0ac58f14700eb6, State: Running, Role: LEADER
I20260929 20:33:13.873785 13773 consensus_queue.cc:237] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64602944447a488d8a0ac58f14700eb6" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 42005 } } peers { permanent_uuid: "6a86a4b292a840b58fc773f99295c67c" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36139 } } peers { permanent_uuid: "e2ed5f2c06fb4efdb4f3e9de12ed66fb" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33195 } }
I20260929 20:33:13.879299 13293 catalog_manager.cc:5495] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64602944447a488d8a0ac58f14700eb6 (127.28.244.129). New cstate: current_term: 1 leader_uuid: "64602944447a488d8a0ac58f14700eb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64602944447a488d8a0ac58f14700eb6" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 42005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a86a4b292a840b58fc773f99295c67c" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2ed5f2c06fb4efdb4f3e9de12ed66fb" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33195 } health_report { overall_health: UNKNOWN } } }
I20260929 20:33:13.899668 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:13.893151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34578 (local address 127.28.244.129:42005)
0929 20:33:13.893434 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:13.893442 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:13.893558 (+ 116us) server_negotiation.cc:407] Connection header received
0929 20:33:13.893833 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:13.893837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:13.894110 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:13.894511 (+ 401us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:33:13.894950 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:13.895695 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:13.897253 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:13.897689 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:13.897856 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:13.898413 (+ 557us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:33:13.899010 (+ 597us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:33:13.899189 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:13.899274 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 20:33:13.899370 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260929 20:33:13.909964 13703 raft_consensus.cc:1261] T fb6da3a61d07493d81b877db2b8d0c5d P 6a86a4b292a840b58fc773f99295c67c [term 1 FOLLOWER]: Refusing update from remote peer 64602944447a488d8a0ac58f14700eb6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:33:13.909964 13569 raft_consensus.cc:1261] T fb6da3a61d07493d81b877db2b8d0c5d P e2ed5f2c06fb4efdb4f3e9de12ed66fb [term 1 FOLLOWER]: Refusing update from remote peer 64602944447a488d8a0ac58f14700eb6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:33:13.911299 13773 consensus_queue.cc:1035] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a86a4b292a840b58fc773f99295c67c" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:33:13.911664 13776 consensus_queue.cc:1035] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2ed5f2c06fb4efdb4f3e9de12ed66fb" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 20:33:13.922292 13616 tablet.cc:2369] T fb6da3a61d07493d81b877db2b8d0c5d P e2ed5f2c06fb4efdb4f3e9de12ed66fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:33:13.926728 13779 mvcc.cc:205] Tried to move back new op lower bound from 7334764519043375104 to 7334764518911528960. Current Snapshot: MvccSnapshot[applied={T|T < 7334764519043375104}]
I20260929 20:33:13.927060 13781 mvcc.cc:205] Tried to move back new op lower bound from 7334764519043375104 to 7334764518911528960. Current Snapshot: MvccSnapshot[applied={T|T < 7334764519043375104}]
I20260929 20:33:13.930147 13780 mvcc.cc:205] Tried to move back new op lower bound from 7334764519043375104 to 7334764518911528960. Current Snapshot: MvccSnapshot[applied={T|T < 7334764519043375104}]
I20260929 20:33:13.944172 13291 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:33:13.944535 13291 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:33:13.949383 13291 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457: Sending DeleteTablet for 3 replicas of tablet fb6da3a61d07493d81b877db2b8d0c5d
I20260929 20:33:13.950830 13415 tablet_service.cc:1514] Processing DeleteTablet for tablet fb6da3a61d07493d81b877db2b8d0c5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:33:13 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:34562
I20260929 20:33:13.951110 13683 tablet_service.cc:1514] Processing DeleteTablet for tablet fb6da3a61d07493d81b877db2b8d0c5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:33:13 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:47974
I20260929 20:33:13.951104 13549 tablet_service.cc:1514] Processing DeleteTablet for tablet fb6da3a61d07493d81b877db2b8d0c5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:33:13 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:41368
I20260929 20:33:13.952658 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 13331
I20260929 20:33:13.961125 13789 tablet_replica.cc:330] T fb6da3a61d07493d81b877db2b8d0c5d P 6a86a4b292a840b58fc773f99295c67c: stopping tablet replica
I20260929 20:33:13.961725 13789 raft_consensus.cc:2229] T fb6da3a61d07493d81b877db2b8d0c5d P 6a86a4b292a840b58fc773f99295c67c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:33:13.962337 13789 raft_consensus.cc:2258] T fb6da3a61d07493d81b877db2b8d0c5d P 6a86a4b292a840b58fc773f99295c67c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:33:13.964135 13790 tablet_replica.cc:330] T fb6da3a61d07493d81b877db2b8d0c5d P e2ed5f2c06fb4efdb4f3e9de12ed66fb: stopping tablet replica
I20260929 20:33:13.964690 13790 raft_consensus.cc:2229] T fb6da3a61d07493d81b877db2b8d0c5d P e2ed5f2c06fb4efdb4f3e9de12ed66fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:33:13.965313 13790 raft_consensus.cc:2258] T fb6da3a61d07493d81b877db2b8d0c5d P e2ed5f2c06fb4efdb4f3e9de12ed66fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:33:13.968134 13789 ts_tablet_manager.cc:1905] T fb6da3a61d07493d81b877db2b8d0c5d P 6a86a4b292a840b58fc773f99295c67c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:33:13.977510 13790 ts_tablet_manager.cc:1905] T fb6da3a61d07493d81b877db2b8d0c5d P e2ed5f2c06fb4efdb4f3e9de12ed66fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:33:13.978597 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:13.954922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34580 (local address 127.28.244.129:42005)
0929 20:33:13.955212 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:13.955222 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:13.955250 (+ 28us) server_negotiation.cc:407] Connection header received
0929 20:33:13.955339 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:13.955348 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:13.955473 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:13.955620 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:33:13.958775 (+ 3155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:13.960747 (+ 1972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:13.963523 (+ 2776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:13.964470 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:13.964629 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:13.969019 (+ 4390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:33:13.969041 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:33:13.969060 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:33:13.969121 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:33:13.973775 (+ 4654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:13.974934 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:13.974946 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:13.974950 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:13.975144 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:13.975613 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:13.975618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:13.975621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:13.976169 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:33:13.976489 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:13.977902 (+ 1413us) server_negotiation.cc:299] Negotiation successful
0929 20:33:13.978196 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260929 20:33:13.979517 13476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:33:13.979930 13789 ts_tablet_manager.cc:1918] T fb6da3a61d07493d81b877db2b8d0c5d P 6a86a4b292a840b58fc773f99295c67c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:33:13.980120 13789 log.cc:1196] T fb6da3a61d07493d81b877db2b8d0c5d P 6a86a4b292a840b58fc773f99295c67c: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/wal/wals/fb6da3a61d07493d81b877db2b8d0c5d
I20260929 20:33:13.980743 13789 ts_tablet_manager.cc:1939] T fb6da3a61d07493d81b877db2b8d0c5d P 6a86a4b292a840b58fc773f99295c67c: Deleting consensus metadata
I20260929 20:33:13.983043 13279 catalog_manager.cc:4841] TS 6a86a4b292a840b58fc773f99295c67c (127.28.244.131:36139): tablet fb6da3a61d07493d81b877db2b8d0c5d (table test-table [id=c50add00c3c44251a7a9f2ee3107454f]) successfully deleted
I20260929 20:33:13.985585 13790 ts_tablet_manager.cc:1918] T fb6da3a61d07493d81b877db2b8d0c5d P e2ed5f2c06fb4efdb4f3e9de12ed66fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:33:13.985976 13790 log.cc:1196] T fb6da3a61d07493d81b877db2b8d0c5d P e2ed5f2c06fb4efdb4f3e9de12ed66fb: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/wal/wals/fb6da3a61d07493d81b877db2b8d0c5d
I20260929 20:33:13.986697 13790 ts_tablet_manager.cc:1939] T fb6da3a61d07493d81b877db2b8d0c5d P e2ed5f2c06fb4efdb4f3e9de12ed66fb: Deleting consensus metadata
I20260929 20:33:13.988902 13280 catalog_manager.cc:4841] TS e2ed5f2c06fb4efdb4f3e9de12ed66fb (127.28.244.130:33195): tablet fb6da3a61d07493d81b877db2b8d0c5d (table test-table [id=c50add00c3c44251a7a9f2ee3107454f]) successfully deleted
I20260929 20:33:14.083160 13788 tablet_replica.cc:330] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6: stopping tablet replica
I20260929 20:33:14.083496 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 13331
I20260929 20:33:14.103242 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 13485
W20260929 20:33:14.103398 13278 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.244.129:42005 (error 108)
I20260929 20:33:14.106081 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:14.105502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:42005 (local address 127.0.0.1:34596)
0929 20:33:14.105717 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:14.105871 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:42005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 20:33:14.106379 13278 catalog_manager.cc:4584] TS 64602944447a488d8a0ac58f14700eb6 (127.28.244.129:42005): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fb6da3a61d07493d81b877db2b8d0c5d: Network error: Client connection negotiation failed: client connection to 127.28.244.129:42005: connect: Connection refused (error 111)
I20260929 20:33:14.120374 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:14.104107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41382 (local address 127.28.244.130:33195)
0929 20:33:14.104415 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:14.104422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:14.104806 (+ 384us) server_negotiation.cc:407] Connection header received
0929 20:33:14.104907 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:14.104913 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:14.104990 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:14.105080 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:33:14.106940 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:14.107969 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:14.109508 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:14.109964 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:14.110085 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:14.113221 (+ 3136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:33:14.113237 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:33:14.113254 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:33:14.113295 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:33:14.116521 (+ 3226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:14.117636 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:14.117646 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:14.117648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:14.117814 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:14.118277 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:14.118281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:14.118284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:14.118797 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:33:14.119056 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:14.119458 (+ 402us) server_negotiation.cc:299] Negotiation successful
0929 20:33:14.119690 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260929 20:33:14.121102 13610 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:33:14.160320 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:14.159740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:42005 (local address 127.0.0.1:34606)
0929 20:33:14.160031 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:14.160151 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:42005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260929 20:33:14.161456 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:14.161078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:42005 (local address 127.0.0.1:34612)
0929 20:33:14.161238 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:14.161333 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:42005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 20:33:14.219175 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 13485
I20260929 20:33:14.222769 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:14.222302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:42005 (local address 127.0.0.1:34614)
0929 20:33:14.222513 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:14.222612 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:42005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 20:33:14.223852 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:14.223457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:42005 (local address 127.0.0.1:34618)
0929 20:33:14.223606 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:14.223693 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:42005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 20:33:14.238159 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 13618
I20260929 20:33:14.253484 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:14.239046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47984 (local address 127.28.244.131:36139)
0929 20:33:14.239247 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:14.239253 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:14.239273 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:33:14.239504 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:14.239509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:14.239578 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:14.239687 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:33:14.241124 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:14.241897 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:14.243374 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:14.243796 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:14.244091 (+ 295us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:14.246922 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:33:14.246939 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:33:14.246955 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:33:14.246993 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:33:14.250014 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:14.250995 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:14.251003 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:14.251006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:14.251158 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:14.251608 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:14.251614 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:14.251622 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:14.252142 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:33:14.252402 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:14.252891 (+ 489us) server_negotiation.cc:299] Negotiation successful
0929 20:33:14.253131 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:33:14.254138 13743 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:33:14.304704 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:14.304159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:42005 (local address 127.0.0.1:34620)
0929 20:33:14.304400 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:14.304500 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:42005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260929 20:33:14.305840 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:14.305426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:42005 (local address 127.0.0.1:34624)
0929 20:33:14.305620 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:14.305704 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:42005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260929 20:33:14.347309 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 13618
I20260929 20:33:14.365475 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 13260
I20260929 20:33:14.366662 13324 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:33:14.506747 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:14.506137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:42005 (local address 127.0.0.1:34630)
0929 20:33:14.506451 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:14.506560 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:42005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260929 20:33:14.506665 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 13260
I20260929 20:33:34.530798 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:35929
--webserver_interface=127.28.244.190
--webserver_port=34105
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:35929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:33:34.924899 13796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:33:34.925216 13796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:33:34.925391 13796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:33:34.933001 13796 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 20:33:34.933111 13796 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 20:33:34.933183 13796 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 20:33:34.933262 13796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:33:34.933336 13796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:33:34.933401 13796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:33:34.933441 13796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:33:34.933490 13796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:33:34.944270 13796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:35929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:35929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=34105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:33:34.945663 13796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:33:34.947081 13796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:33:34.960055 13802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:33:34.960023 13804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:33:34.960392 13801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:33:34.961285 13796 server_base.cc:1034] running on GCE node
I20260929 20:33:34.962167 13796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:33:34.963773 13796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:33:34.965077 13796 hybrid_clock.cc:648] HybridClock initialized: now 1790714014965015 us; error 43 us; skew 500 ppm
I20260929 20:33:34.965435 13796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:33:34 dist-test-slave-p29l krb5kdc[12696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714014, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:33:34.968737 13796 init.cc:377] Logged in from keytab as kudu/127.28.244.190@KRBTEST.COM (short username kudu)
I20260929 20:33:34.970434 13796 webserver.cc:460] Webserver started at http://127.28.244.190:34105/ using document root <none> and password file <none>
I20260929 20:33:34.971199 13796 fs_manager.cc:359] Metadata directory not provided
I20260929 20:33:34.971338 13796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:33:34.977155 13796 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 20:33:34.980093 13811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:33:34.981544 13796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:33:34.981740 13796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "391fcaa2e0d84f20b5f803c250eea457"
format_stamp: "Formatted at 2026-09-29 20:32:48 on dist-test-slave-p29l"
server_key: "4b21e2e5d948004ed4c01ce708b1f639"
server_key_iv: "3ab699e47e976be32682fc4d568d3e32"
server_key_version: "encryptionkey@0"
I20260929 20:33:34.982748 13796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:33:35.002056 13796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:33:35.006557 13796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:33:35.006868 13796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:33:35.026422 13796 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:35929
I20260929 20:33:35.026496 13863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:35929 every 8 connection(s)
I20260929 20:33:35.028060 13796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:33:35.033383 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 13796
I20260929 20:33:35.034382 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.129:42005
--local_ip_for_outbound_sockets=127.28.244.129
--tserver_master_addrs=127.28.244.190:35929
--webserver_port=43257
--webserver_interface=127.28.244.129
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:33:35.036108 13864 sys_catalog.cc:263] Verifying existing consensus state
I20260929 20:33:35.041482 13864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457: Bootstrap starting.
I20260929 20:33:35.063467 13864 log.cc:824] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457: Log is configured to *not* fsync() on all Append() calls
I20260929 20:33:35.080818 13864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:33:35.081430 13864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457: Bootstrap complete.
I20260929 20:33:35.091559 13864 raft_consensus.cc:348] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391fcaa2e0d84f20b5f803c250eea457" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 35929 } }
I20260929 20:33:35.092564 13864 raft_consensus.cc:729] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 391fcaa2e0d84f20b5f803c250eea457, State: Initialized, Role: FOLLOWER
I20260929 20:33:35.093266 13864 consensus_queue.cc:260] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391fcaa2e0d84f20b5f803c250eea457" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 35929 } }
I20260929 20:33:35.093519 13864 raft_consensus.cc:388] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:33:35.093636 13864 raft_consensus.cc:482] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:33:35.093837 13864 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [term 2 FOLLOWER]: Advancing to term 3
I20260929 20:33:35.098039 13864 raft_consensus.cc:504] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391fcaa2e0d84f20b5f803c250eea457" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 35929 } }
I20260929 20:33:35.098658 13864 leader_election.cc:302] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 391fcaa2e0d84f20b5f803c250eea457; no voters:
I20260929 20:33:35.099447 13864 leader_election.cc:288] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 20:33:35.099593 13868 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 20:33:35.100886 13868 raft_consensus.cc:686] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [term 3 LEADER]: Becoming Leader. State: Replica: 391fcaa2e0d84f20b5f803c250eea457, State: Running, Role: LEADER
I20260929 20:33:35.101650 13868 consensus_queue.cc:237] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391fcaa2e0d84f20b5f803c250eea457" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 35929 } }
I20260929 20:33:35.101958 13864 sys_catalog.cc:564] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:33:35.104400 13870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 391fcaa2e0d84f20b5f803c250eea457. Latest consensus state: current_term: 3 leader_uuid: "391fcaa2e0d84f20b5f803c250eea457" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391fcaa2e0d84f20b5f803c250eea457" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 35929 } } }
I20260929 20:33:35.104842 13870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [sys.catalog]: This master's current role is: LEADER
I20260929 20:33:35.105448 13869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "391fcaa2e0d84f20b5f803c250eea457" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391fcaa2e0d84f20b5f803c250eea457" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 35929 } } }
I20260929 20:33:35.105868 13869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457 [sys.catalog]: This master's current role is: LEADER
I20260929 20:33:35.115777 13876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:33:35.125435 13876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:33:35.127592 13876 catalog_manager.cc:1259] Loaded cluster ID: bdece67bdd2a4aacbc18219848e5f87a
I20260929 20:33:35.127833 13876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:33:35.132315 13885 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 20:33:35.133224 13876 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:33:35.134616 13876 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457: Loaded TSK: 0
I20260929 20:33:35.134788 13876 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 391fcaa2e0d84f20b5f803c250eea457: Loaded TSK: 1
F20260929 20:33:35.135690 13876 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe68b7419ad google::LogMessage::Fail() at ??:0
@ 0x7fe68b74488a google::LogMessage::SendToLog() at ??:0
@ 0x7fe68b7414de google::LogMessage::Flush() at ??:0
@ 0x7fe68b742a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe68f4a57d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fe68f49473e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fe6a11741c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fe6a1173783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fe6a11ff2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fe6a11ff0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fe6a12495e9 std::function<>::operator()() at ??:0
@ 0x7fe6a1177cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fe6a1175c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe6a11fd610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fe6a11fd439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fe68e38d345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe68e39091c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fe68e390799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fe68e36beaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe6916456db start_thread at ??:0
@ 0x7fe68a5db61f clone at ??:0
W20260929 20:33:35.443949 13866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:33:35.444307 13866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:33:35.444481 13866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:33:35.452271 13866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:33:35.452450 13866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:33:35.452629 13866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.129
I20260929 20:33:35.463281 13866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.129:42005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.244.129
--webserver_port=43257
--tserver_master_addrs=127.28.244.190:35929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.129
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:33:35.464818 13866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:33:35.466271 13866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:33:35.480860 13911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:33:35.480834 13912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:33:35.480834 13914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:33:35.482005 13866 server_base.cc:1034] running on GCE node
I20260929 20:33:35.482715 13866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:33:35.484012 13866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:33:35.485304 13866 hybrid_clock.cc:648] HybridClock initialized: now 1790714015485242 us; error 75 us; skew 500 ppm
I20260929 20:33:35.485798 13866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:33:35 dist-test-slave-p29l krb5kdc[12696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714015, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:33:35.489346 13866 init.cc:377] Logged in from keytab as kudu/127.28.244.129@KRBTEST.COM (short username kudu)
I20260929 20:33:35.491005 13866 webserver.cc:460] Webserver started at http://127.28.244.129:43257/ using document root <none> and password file <none>
I20260929 20:33:35.491776 13866 fs_manager.cc:359] Metadata directory not provided
I20260929 20:33:35.491941 13866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:33:35.497929 13866 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 20:33:35.501106 13921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:33:35.502755 13866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:33:35.502940 13866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/wal
uuid: "64602944447a488d8a0ac58f14700eb6"
format_stamp: "Formatted at 2026-09-29 20:32:49 on dist-test-slave-p29l"
server_key: "971a8af187bd30975ba0755262c6c45d"
server_key_iv: "375b1677fced77eb8228f478c9e3bf40"
server_key_version: "encryptionkey@0"
I20260929 20:33:35.504048 13866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:33:35.536657 13866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:33:35.542160 13866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:33:35.542469 13866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:33:35.543537 13866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:33:35.545536 13928 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260929 20:33:35.551783 13866 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 20:33:35.552024 13866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260929 20:33:35.552199 13866 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 20:33:35.556077 13866 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 20:33:35.556210 13866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.004s sys 0.000s
I20260929 20:33:35.556816 13928 tablet_bootstrap.cc:492] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6: Bootstrap starting.
I20260929 20:33:35.590019 13928 log.cc:824] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6: Log is configured to *not* fsync() on all Append() calls
I20260929 20:33:35.601624 13866 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.129:42005
I20260929 20:33:35.601724 14036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.129:42005 every 8 connection(s)
I20260929 20:33:35.603076 13866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
I20260929 20:33:35.604555 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 13866
I20260929 20:33:35.605168 13928 tablet_bootstrap.cc:492] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:33:35.605690 13928 tablet_bootstrap.cc:492] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6: Bootstrap complete.
I20260929 20:33:35.605688 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.130:33195
--local_ip_for_outbound_sockets=127.28.244.130
--tserver_master_addrs=127.28.244.190:35929
--webserver_port=42909
--webserver_interface=127.28.244.130
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:33:35.606956 13928 ts_tablet_manager.cc:1397] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6: Time spent bootstrapping tablet: real 0.051s user 0.042s sys 0.005s
I20260929 20:33:35.622524 14039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:35.608745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.129:37857)
0929 20:33:35.620588 (+ 11843us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:35.620992 (+ 404us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11462,"thread_start_us":11164,"threads_started":1}
W20260929 20:33:35.623379 14037 heartbeater.cc:644] Failed to heartbeat to 127.28.244.190:35929 (0 consecutive failures): Network error: Failed to ping master at 127.28.244.190:35929: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
I20260929 20:33:35.622641 13928 raft_consensus.cc:348] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64602944447a488d8a0ac58f14700eb6" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 42005 } } peers { permanent_uuid: "6a86a4b292a840b58fc773f99295c67c" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36139 } } peers { permanent_uuid: "e2ed5f2c06fb4efdb4f3e9de12ed66fb" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33195 } }
I20260929 20:33:35.624125 13928 raft_consensus.cc:729] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64602944447a488d8a0ac58f14700eb6, State: Initialized, Role: FOLLOWER
I20260929 20:33:35.624655 14039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:35.624165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.129:51947)
0929 20:33:35.624419 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:35.624508 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260929 20:33:35.624867 13928 consensus_queue.cc:260] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64602944447a488d8a0ac58f14700eb6" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 42005 } } peers { permanent_uuid: "6a86a4b292a840b58fc773f99295c67c" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36139 } } peers { permanent_uuid: "e2ed5f2c06fb4efdb4f3e9de12ed66fb" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33195 } }
I20260929 20:33:35.626386 14041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:35.625226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.129:60221)
0929 20:33:35.625888 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:35.626109 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":273,"threads_started":1}
W20260929 20:33:35.626770 14037 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 20:33:35.627890 14041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:35.627439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.129:48905)
0929 20:33:35.627608 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:35.627704 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 20:33:35.627940 13928 ts_tablet_manager.cc:1428] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6: Time spent starting tablet: real 0.021s user 0.022s sys 0.000s
I20260929 20:33:35.629173 14041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:35.628769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.129:41877)
0929 20:33:35.628958 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:35.629050 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260929 20:33:35.965767 14040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:33:35.966120 14040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:33:35.966292 14040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:33:35.973879 14040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:33:35.974048 14040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:33:35.974248 14040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.130
I20260929 20:33:35.984660 14040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.130:33195
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.244.130
--webserver_port=42909
--tserver_master_addrs=127.28.244.190:35929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.130
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:33:35.986109 14040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:33:35.987460 14040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:33:36.002223 14047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:33:36.002135 14048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:33:36.002199 14050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:33:36.005666 14040 server_base.cc:1034] running on GCE node
I20260929 20:33:36.006402 14040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:33:36.007513 14040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:33:36.008796 14040 hybrid_clock.cc:648] HybridClock initialized: now 1790714016008745 us; error 41 us; skew 500 ppm
I20260929 20:33:36.009167 14040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:33:36 dist-test-slave-p29l krb5kdc[12696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714016, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:33:36.012797 14040 init.cc:377] Logged in from keytab as kudu/127.28.244.130@KRBTEST.COM (short username kudu)
I20260929 20:33:36.014380 14040 webserver.cc:460] Webserver started at http://127.28.244.130:42909/ using document root <none> and password file <none>
I20260929 20:33:36.015113 14040 fs_manager.cc:359] Metadata directory not provided
I20260929 20:33:36.015220 14040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:33:36.021235 14040 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 20:33:36.024500 14057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:33:36.026147 14040 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:33:36.026332 14040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/wal
uuid: "e2ed5f2c06fb4efdb4f3e9de12ed66fb"
format_stamp: "Formatted at 2026-09-29 20:32:49 on dist-test-slave-p29l"
server_key: "aa0c4959742f053c7875eb4cebcb0a40"
server_key_iv: "f208ac299da3e23631a51bbe7394b0f4"
server_key_version: "encryptionkey@0"
I20260929 20:33:36.027428 14040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:33:36.054729 14040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:33:36.060068 14040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:33:36.060350 14040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:33:36.061461 14040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:33:36.062983 14040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:33:36.063083 14040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:33:36.063177 14040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:33:36.063236 14040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:33:36.100284 14040 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.130:33195
I20260929 20:33:36.100387 14170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.130:33195 every 8 connection(s)
I20260929 20:33:36.101984 14040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
I20260929 20:33:36.107281 14173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:36.105245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.130:50657)
0929 20:33:36.106079 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:36.106378 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":309,"threads_started":1}
W20260929 20:33:36.108083 14171 heartbeater.cc:644] Failed to heartbeat to 127.28.244.190:35929 (0 consecutive failures): Network error: Failed to ping master at 127.28.244.190:35929: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
I20260929 20:33:36.109081 14173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:36.108689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.130:57413)
0929 20:33:36.108883 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:36.108966 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 20:33:36.109956 14173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:36.109652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.130:55429)
0929 20:33:36.109777 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:36.109851 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 20:33:36.110241 14171 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 20:33:36.112159 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 14040
I20260929 20:33:36.112773 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.131:36139
--local_ip_for_outbound_sockets=127.28.244.131
--tserver_master_addrs=127.28.244.190:35929
--webserver_port=45243
--webserver_interface=127.28.244.131
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:33:36.466924 14174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:33:36.467227 14174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:33:36.467324 14174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:33:36.474810 14174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:33:36.474951 14174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:33:36.475067 14174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.131
I20260929 20:33:36.485412 14174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.131:36139
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.244.131
--webserver_port=45243
--tserver_master_addrs=127.28.244.190:35929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.131
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:33:36.486860 14174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:33:36.488277 14174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:33:36.501833 14179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:33:36.501896 14180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:33:36.501870 14182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:33:36.502858 14174 server_base.cc:1034] running on GCE node
I20260929 20:33:36.503480 14174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:33:36.504530 14174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:33:36.505805 14174 hybrid_clock.cc:648] HybridClock initialized: now 1790714016505730 us; error 66 us; skew 500 ppm
I20260929 20:33:36.506160 14174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:33:36 dist-test-slave-p29l krb5kdc[12696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714016, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:33:36.509935 14174 init.cc:377] Logged in from keytab as kudu/127.28.244.131@KRBTEST.COM (short username kudu)
I20260929 20:33:36.511776 14174 webserver.cc:460] Webserver started at http://127.28.244.131:45243/ using document root <none> and password file <none>
I20260929 20:33:36.512632 14174 fs_manager.cc:359] Metadata directory not provided
I20260929 20:33:36.512836 14174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:33:36.520339 14174 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 20:33:36.523577 14189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:33:36.525216 14174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:33:36.525552 14174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/wal
uuid: "6a86a4b292a840b58fc773f99295c67c"
format_stamp: "Formatted at 2026-09-29 20:32:50 on dist-test-slave-p29l"
server_key: "3324456691340329677eae6bf32be348"
server_key_iv: "79e5be5939164f1c12c25b5a58dda085"
server_key_version: "encryptionkey@0"
I20260929 20:33:36.526609 14174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:33:36.552445 14174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:33:36.557271 14174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:33:36.557574 14174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:33:36.558661 14174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:33:36.560180 14174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:33:36.560316 14174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:33:36.560452 14174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:33:36.560544 14174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:33:36.595127 14174 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.131:36139
I20260929 20:33:36.595216 14303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.131:36139 every 8 connection(s)
I20260929 20:33:36.596660 14174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
I20260929 20:33:36.599534 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 14174
I20260929 20:33:36.602768 14306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:36.599914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.131:33827)
0929 20:33:36.600832 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:36.601372 (+ 540us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":303,"threads_started":1}
W20260929 20:33:36.603835 14304 heartbeater.cc:644] Failed to heartbeat to 127.28.244.190:35929 (0 consecutive failures): Network error: Failed to ping master at 127.28.244.190:35929: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
I20260929 20:33:36.604804 14306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:36.604458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.131:35793)
0929 20:33:36.604618 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:36.604694 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 20:33:36.605693 14306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:36.605322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.131:58305)
0929 20:33:36.605509 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:36.605581 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260929 20:33:36.605933 14304 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 20:33:36.631019 14308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:36.630112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.129:52757)
0929 20:33:36.630627 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:36.630714 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":260,"threads_started":1}
I20260929 20:33:36.971127 14309 raft_consensus.cc:482] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:33:36.971450 14309 raft_consensus.cc:504] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64602944447a488d8a0ac58f14700eb6" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 42005 } } peers { permanent_uuid: "6a86a4b292a840b58fc773f99295c67c" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36139 } } peers { permanent_uuid: "e2ed5f2c06fb4efdb4f3e9de12ed66fb" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33195 } }
I20260929 20:33:36.973134 14309 leader_election.cc:288] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a86a4b292a840b58fc773f99295c67c (127.28.244.131:36139), e2ed5f2c06fb4efdb4f3e9de12ed66fb (127.28.244.130:33195)
Sep 29 20:33:36 dist-test-slave-p29l krb5kdc[12696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714015, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for kudu/127.28.244.130@KRBTEST.COM
Sep 29 20:33:36 dist-test-slave-p29l krb5kdc[12696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714015, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for kudu/127.28.244.131@KRBTEST.COM
I20260929 20:33:36.998720 14312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:36.973768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:41823 (local address 127.28.244.130:33195)
0929 20:33:36.976043 (+ 2275us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:36.976057 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:36.976093 (+ 36us) server_negotiation.cc:407] Connection header received
0929 20:33:36.978283 (+ 2190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:36.978296 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:36.978842 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:36.979233 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:33:36.981107 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:36.982312 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:36.984295 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:36.984931 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:36.985149 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:36.989396 (+ 4247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:33:36.989405 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:33:36.989442 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:33:36.989501 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:33:36.993697 (+ 4196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:36.994975 (+ 1278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:36.994985 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:36.994994 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:36.995236 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:36.995841 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:36.995847 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:36.995849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:36.996571 (+ 722us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:33:36.996766 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:36.997866 (+ 1100us) server_negotiation.cc:299] Negotiation successful
0929 20:33:36.998279 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2084,"thread_start_us":1837,"threads_started":1}
I20260929 20:33:36.998714 14311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:36.973747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:52731 (local address 127.28.244.131:36139)
0929 20:33:36.975451 (+ 1704us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:36.975468 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:36.975914 (+ 446us) server_negotiation.cc:407] Connection header received
0929 20:33:36.978293 (+ 2379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:36.978302 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:36.978842 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:36.979233 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:33:36.981134 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:36.982308 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:36.984260 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:36.984930 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:36.985160 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:36.989653 (+ 4493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:33:36.989665 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:33:36.989681 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:33:36.989732 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:33:36.993697 (+ 3965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:36.994975 (+ 1278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:36.994985 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:36.994994 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:36.995236 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:36.995804 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:36.995832 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:36.995835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:36.996571 (+ 736us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:33:36.996766 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:36.997829 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0929 20:33:36.998279 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1452,"thread_start_us":1198,"threads_started":1}
I20260929 20:33:36.998733 14308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:36.973467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:33195 (local address 127.28.244.129:41823)
0929 20:33:36.973628 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:36.973677 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 20:33:36.974385 (+ 708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:33:36.979369 (+ 4984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:33:36.979378 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:33:36.980142 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:33:36.980881 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:36.980912 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:36.982537 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:36.982541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:36.984120 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:36.984138 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:36.985201 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:36.985209 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:36.985499 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:36.986340 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:33:36.986357 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:33:36.989067 (+ 2710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:33:36.993886 (+ 4819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:33:36.993896 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:33:36.993914 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:33:36.994796 (+ 882us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:33:36.995378 (+ 582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:33:36.995383 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:33:36.995386 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:33:36.995671 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:33:36.996723 (+ 1052us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:33:36.996732 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:33:36.997185 (+ 453us) client_negotiation.cc:769] Sending connection context
0929 20:33:36.997523 (+ 338us) client_negotiation.cc:240] Negotiation successful
0929 20:33:36.998162 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":173,"spinlock_wait_cycles":19968}
I20260929 20:33:36.998745 14310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:36.973466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:36139 (local address 127.28.244.129:52731)
0929 20:33:36.975742 (+ 2276us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:36.975763 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:33:36.975981 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:33:36.979899 (+ 3918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:33:36.979903 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:33:36.980406 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:33:36.980882 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:36.980903 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:36.982529 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:36.982534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:36.984092 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:36.984107 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:36.985148 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:36.985155 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:36.985503 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:36.986311 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:33:36.986346 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:33:36.989353 (+ 3007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:33:36.993886 (+ 4533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:33:36.993896 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:33:36.993914 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:33:36.994796 (+ 882us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:33:36.995378 (+ 582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:33:36.995383 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:33:36.995386 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:33:36.995671 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:33:36.996723 (+ 1052us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:33:36.996732 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:33:36.997185 (+ 453us) client_negotiation.cc:769] Sending connection context
0929 20:33:36.997523 (+ 338us) client_negotiation.cc:240] Negotiation successful
0929 20:33:36.998275 (+ 752us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2145,"mutex_wait_us":640,"spinlock_wait_cycles":60032,"thread_start_us":1754,"threads_started":1}
W20260929 20:33:37.002466 13925 leader_election.cc:341] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2ed5f2c06fb4efdb4f3e9de12ed66fb (127.28.244.130:33195): Not found: Tablet not found: fb6da3a61d07493d81b877db2b8d0c5d
W20260929 20:33:37.002763 13924 leader_election.cc:341] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6a86a4b292a840b58fc773f99295c67c (127.28.244.131:36139): Not found: Tablet not found: fb6da3a61d07493d81b877db2b8d0c5d
I20260929 20:33:37.002990 13924 leader_election.cc:302] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64602944447a488d8a0ac58f14700eb6; no voters: 6a86a4b292a840b58fc773f99295c67c, e2ed5f2c06fb4efdb4f3e9de12ed66fb
I20260929 20:33:37.003432 14309 raft_consensus.cc:2726] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 20:33:37.111857 14313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:37.110887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.130:54807)
0929 20:33:37.111396 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:37.111552 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":266,"threads_started":1}
I20260929 20:33:37.607515 14314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:37.606566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.131:33999)
0929 20:33:37.607168 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:37.607297 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":257,"threads_started":1}
I20260929 20:33:37.633018 14315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:37.632176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.129:48905)
0929 20:33:37.632664 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:37.632808 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":230,"threads_started":1}
I20260929 20:33:38.113708 14316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:38.112972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.130:44235)
0929 20:33:38.113448 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:38.113545 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":273,"threads_started":1}
I20260929 20:33:38.609411 14317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:38.608530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.131:45013)
0929 20:33:38.609102 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:38.609189 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":322,"threads_started":1}
I20260929 20:33:38.634958 14318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:38.634088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.129:55195)
0929 20:33:38.634623 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:38.634712 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":243,"threads_started":1}
I20260929 20:33:38.964843 14319 raft_consensus.cc:482] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:33:38.965090 14319 raft_consensus.cc:504] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64602944447a488d8a0ac58f14700eb6" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 42005 } } peers { permanent_uuid: "6a86a4b292a840b58fc773f99295c67c" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36139 } } peers { permanent_uuid: "e2ed5f2c06fb4efdb4f3e9de12ed66fb" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33195 } }
I20260929 20:33:38.965823 14319 leader_election.cc:288] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a86a4b292a840b58fc773f99295c67c (127.28.244.131:36139), e2ed5f2c06fb4efdb4f3e9de12ed66fb (127.28.244.130:33195)
W20260929 20:33:38.966955 13925 leader_election.cc:341] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2ed5f2c06fb4efdb4f3e9de12ed66fb (127.28.244.130:33195): Not found: Tablet not found: fb6da3a61d07493d81b877db2b8d0c5d
W20260929 20:33:38.967229 13924 leader_election.cc:341] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6a86a4b292a840b58fc773f99295c67c (127.28.244.131:36139): Not found: Tablet not found: fb6da3a61d07493d81b877db2b8d0c5d
I20260929 20:33:38.967360 13924 leader_election.cc:302] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64602944447a488d8a0ac58f14700eb6; no voters: 6a86a4b292a840b58fc773f99295c67c, e2ed5f2c06fb4efdb4f3e9de12ed66fb
I20260929 20:33:38.967700 14319 raft_consensus.cc:2726] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 20:33:39.115607 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:39.114782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.130:56157)
0929 20:33:39.115329 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:39.115431 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":293,"threads_started":1}
I20260929 20:33:39.611078 14321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:39.610350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.131:57129)
0929 20:33:39.610822 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:39.610916 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":252,"threads_started":1}
I20260929 20:33:39.637001 14322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:39.636202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.129:40427)
0929 20:33:39.636752 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:39.636835 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":311,"threads_started":1}
I20260929 20:33:40.117331 14323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:40.116601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.130:41269)
0929 20:33:40.117054 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:40.117142 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":227,"threads_started":1}
I20260929 20:33:40.613240 14324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:40.612381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.131:40489)
0929 20:33:40.612925 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:40.613029 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":251,"threads_started":1}
I20260929 20:33:40.639259 14325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:40.638325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.129:34849)
0929 20:33:40.638900 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:40.638998 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":342,"threads_started":1}
I20260929 20:33:41.119122 14326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:41.118342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.130:43167)
0929 20:33:41.118853 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:41.118941 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":249,"threads_started":1}
I20260929 20:33:41.615113 14327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:41.614310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.131:58343)
0929 20:33:41.614835 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:41.614939 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":275,"threads_started":1}
I20260929 20:33:41.641346 14328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:41.640557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.129:51579)
0929 20:33:41.641066 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:41.641171 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":257,"threads_started":1}
I20260929 20:33:42.028018 14329 raft_consensus.cc:482] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:33:42.028286 14329 raft_consensus.cc:504] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64602944447a488d8a0ac58f14700eb6" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 42005 } } peers { permanent_uuid: "6a86a4b292a840b58fc773f99295c67c" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36139 } } peers { permanent_uuid: "e2ed5f2c06fb4efdb4f3e9de12ed66fb" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33195 } }
I20260929 20:33:42.029049 14329 leader_election.cc:288] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a86a4b292a840b58fc773f99295c67c (127.28.244.131:36139), e2ed5f2c06fb4efdb4f3e9de12ed66fb (127.28.244.130:33195)
W20260929 20:33:42.030143 13924 leader_election.cc:341] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6a86a4b292a840b58fc773f99295c67c (127.28.244.131:36139): Not found: Tablet not found: fb6da3a61d07493d81b877db2b8d0c5d
W20260929 20:33:42.030469 13925 leader_election.cc:341] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2ed5f2c06fb4efdb4f3e9de12ed66fb (127.28.244.130:33195): Not found: Tablet not found: fb6da3a61d07493d81b877db2b8d0c5d
I20260929 20:33:42.030616 13925 leader_election.cc:302] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64602944447a488d8a0ac58f14700eb6; no voters: 6a86a4b292a840b58fc773f99295c67c, e2ed5f2c06fb4efdb4f3e9de12ed66fb
I20260929 20:33:42.030900 14329 raft_consensus.cc:2726] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 20:33:42.121161 14330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:42.120254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.130:40359)
0929 20:33:42.120793 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:42.120899 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":262,"threads_started":1}
I20260929 20:33:42.617187 14331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:42.616401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.131:52549)
0929 20:33:42.616900 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:42.616990 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":253,"threads_started":1}
I20260929 20:33:42.643332 14332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:42.642501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.129:36089)
0929 20:33:42.643005 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:42.643112 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":261,"threads_started":1}
I20260929 20:33:43.123299 14333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:43.122396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.130:49455)
0929 20:33:43.122996 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:43.123097 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":265,"threads_started":1}
I20260929 20:33:43.619024 14334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:43.618175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.131:46655)
0929 20:33:43.618724 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:43.618819 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":253,"threads_started":1}
I20260929 20:33:43.645277 14335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:43.644442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.129:60255)
0929 20:33:43.644985 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:43.645102 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":295,"threads_started":1}
I20260929 20:33:44.125274 14336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:44.124501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.130:38421)
0929 20:33:44.124972 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:44.125072 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":238,"threads_started":1}
I20260929 20:33:44.620765 14337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:44.620018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.131:57589)
0929 20:33:44.620501 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:44.620581 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":249,"threads_started":1}
I20260929 20:33:44.647655 14338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:44.646425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.129:45191)
0929 20:33:44.647274 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:44.647396 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":577,"threads_started":1}
I20260929 20:33:45.127247 14339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:45.126423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.130:54575)
0929 20:33:45.126967 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:45.127074 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":241,"threads_started":1}
I20260929 20:33:45.622591 14340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:45.621827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.131:49415)
0929 20:33:45.622326 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:45.622425 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":249,"threads_started":1}
I20260929 20:33:45.649607 14341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:45.648784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.129:45003)
0929 20:33:45.649341 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:45.649434 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":326,"threads_started":1}
I20260929 20:33:45.945591 14342 raft_consensus.cc:482] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:33:45.945849 14342 raft_consensus.cc:504] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64602944447a488d8a0ac58f14700eb6" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 42005 } } peers { permanent_uuid: "6a86a4b292a840b58fc773f99295c67c" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36139 } } peers { permanent_uuid: "e2ed5f2c06fb4efdb4f3e9de12ed66fb" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33195 } }
I20260929 20:33:45.946540 14342 leader_election.cc:288] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a86a4b292a840b58fc773f99295c67c (127.28.244.131:36139), e2ed5f2c06fb4efdb4f3e9de12ed66fb (127.28.244.130:33195)
W20260929 20:33:45.947758 13924 leader_election.cc:341] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6a86a4b292a840b58fc773f99295c67c (127.28.244.131:36139): Not found: Tablet not found: fb6da3a61d07493d81b877db2b8d0c5d
W20260929 20:33:45.948004 13925 leader_election.cc:341] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2ed5f2c06fb4efdb4f3e9de12ed66fb (127.28.244.130:33195): Not found: Tablet not found: fb6da3a61d07493d81b877db2b8d0c5d
I20260929 20:33:45.948102 13925 leader_election.cc:302] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64602944447a488d8a0ac58f14700eb6; no voters: 6a86a4b292a840b58fc773f99295c67c, e2ed5f2c06fb4efdb4f3e9de12ed66fb
I20260929 20:33:45.948347 14342 raft_consensus.cc:2726] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 20:33:46.129073 14343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:46.128288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.130:59365)
0929 20:33:46.128768 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:46.128864 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":259,"threads_started":1}
I20260929 20:33:46.624568 14344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:46.623674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.131:33697)
0929 20:33:46.624269 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:46.624367 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":353,"threads_started":1}
I20260929 20:33:46.651578 14345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:46.650741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.129:52895)
0929 20:33:46.651257 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:46.651384 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":274,"threads_started":1}
I20260929 20:33:47.131213 14346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:47.130217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.130:45017)
0929 20:33:47.130813 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:47.130960 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":273,"threads_started":1}
I20260929 20:33:47.626462 14347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:47.625611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.131:34159)
0929 20:33:47.626173 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:47.626288 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":282,"threads_started":1}
I20260929 20:33:47.653599 14348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:47.652739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.129:44997)
0929 20:33:47.653317 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:47.653425 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":308,"threads_started":1}
I20260929 20:33:48.133199 14349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:48.132388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.130:55859)
0929 20:33:48.132909 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:48.133001 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":232,"threads_started":1}
I20260929 20:33:48.297156 14350 raft_consensus.cc:482] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:33:48.297379 14350 raft_consensus.cc:504] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64602944447a488d8a0ac58f14700eb6" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 42005 } } peers { permanent_uuid: "6a86a4b292a840b58fc773f99295c67c" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 36139 } } peers { permanent_uuid: "e2ed5f2c06fb4efdb4f3e9de12ed66fb" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 33195 } }
I20260929 20:33:48.298159 14350 leader_election.cc:288] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a86a4b292a840b58fc773f99295c67c (127.28.244.131:36139), e2ed5f2c06fb4efdb4f3e9de12ed66fb (127.28.244.130:33195)
W20260929 20:33:48.299266 13924 leader_election.cc:341] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6a86a4b292a840b58fc773f99295c67c (127.28.244.131:36139): Not found: Tablet not found: fb6da3a61d07493d81b877db2b8d0c5d
W20260929 20:33:48.299476 13925 leader_election.cc:341] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2ed5f2c06fb4efdb4f3e9de12ed66fb (127.28.244.130:33195): Not found: Tablet not found: fb6da3a61d07493d81b877db2b8d0c5d
I20260929 20:33:48.299626 13925 leader_election.cc:302] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64602944447a488d8a0ac58f14700eb6; no voters: 6a86a4b292a840b58fc773f99295c67c, e2ed5f2c06fb4efdb4f3e9de12ed66fb
I20260929 20:33:48.299993 14350 raft_consensus.cc:2726] T fb6da3a61d07493d81b877db2b8d0c5d P 64602944447a488d8a0ac58f14700eb6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 20:33:48.628468 14351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:48.627652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.131:50735)
0929 20:33:48.628200 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:48.628282 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":306,"threads_started":1}
I20260929 20:33:48.655586 14352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:48.654693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.129:52029)
0929 20:33:48.655232 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:48.655363 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":265,"threads_started":1}
I20260929 20:33:49.135046 14353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:49.134212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.130:42277)
0929 20:33:49.134724 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:49.134866 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":266,"threads_started":1}
I20260929 20:33:49.630355 14354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:49.629573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.131:33199)
0929 20:33:49.630058 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:49.630182 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":238,"threads_started":1}
I20260929 20:33:49.657783 14355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:49.656859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.129:56651)
0929 20:33:49.657462 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:49.657554 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":315,"threads_started":1}
I20260929 20:33:50.136734 14356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:50.136003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.130:43561)
0929 20:33:50.136449 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:50.136528 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":230,"threads_started":1}
I20260929 20:33:50.632426 14357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:50.631411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.131:41913)
0929 20:33:50.632057 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:50.632165 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":315,"threads_started":1}
I20260929 20:33:50.660048 14358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:50.659118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.129:33567)
0929 20:33:50.659678 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:50.659766 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":302,"threads_started":1}
I20260929 20:33:51.138837 14359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:51.138036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.130:53011)
0929 20:33:51.138538 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:51.138654 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":271,"threads_started":1}
I20260929 20:33:51.612610 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 13866
I20260929 20:33:51.633955 14360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:51.613904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40168 (local address 127.28.244.129:42005)
0929 20:33:51.614731 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:51.614742 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:51.614877 (+ 135us) server_negotiation.cc:407] Connection header received
0929 20:33:51.615082 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:51.615090 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:51.615337 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:51.615498 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:33:51.617139 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:51.618187 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:51.619521 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:51.620249 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:51.620362 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:51.627433 (+ 7071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:33:51.627451 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:33:51.627467 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:33:51.627515 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:33:51.630778 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:51.631581 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:51.631588 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:51.631591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:51.631776 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:51.632216 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:51.632221 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:51.632224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:51.632703 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:33:51.633012 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:51.633319 (+ 307us) server_negotiation.cc:299] Negotiation successful
0929 20:33:51.633570 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":360,"threads_started":1}
I20260929 20:33:51.634552 14361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:51.633666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:35929 (local address 127.28.244.131:39517)
0929 20:33:51.634204 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:51.634352 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.190:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":281,"threads_started":1}
I20260929 20:33:51.635442 14032 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:33:51.727636 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 13866
I20260929 20:33:51.746968 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 14040
I20260929 20:33:51.762128 14364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:51.747861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49954 (local address 127.28.244.130:33195)
0929 20:33:51.748393 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:51.748398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:51.748415 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:33:51.748473 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:51.748477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:51.748537 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:51.748630 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:33:51.750040 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:51.751032 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:51.752377 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:51.752949 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:51.753074 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:51.755883 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:33:51.755891 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:33:51.755896 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:33:51.755940 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:33:51.759079 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:51.759950 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:51.759954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:51.759963 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:51.760108 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:51.760569 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:51.760574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:51.760576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:51.760896 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:33:51.761128 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:51.761473 (+ 345us) server_negotiation.cc:299] Negotiation successful
0929 20:33:51.761662 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":298,"threads_started":1}
I20260929 20:33:51.762880 14166 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:33:51.850353 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 14040
I20260929 20:33:51.874004 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 14174
I20260929 20:33:51.891788 14366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:51.874979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60216 (local address 127.28.244.131:36139)
0929 20:33:51.875868 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:51.875877 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:51.875900 (+ 23us) server_negotiation.cc:407] Connection header received
0929 20:33:51.875973 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:51.875981 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:51.876082 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:51.876206 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:33:51.877597 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:51.878941 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:51.880510 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:51.881365 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:51.881515 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:51.884435 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:33:51.884445 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:33:51.884451 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:33:51.884503 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:33:51.888581 (+ 4078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:51.889401 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:51.889410 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:51.889415 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:51.889597 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:51.890034 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:51.890041 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:51.890045 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:51.890443 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:33:51.890743 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:51.891018 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 20:33:51.891286 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":515,"threads_started":1}
I20260929 20:33:51.892690 14299 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:33:51.977625 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 14174
2026-09-29T20:33:51Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65134 ms)
[----------] 30 tests from SecurityITest (310740 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:33:52 dist-test-slave-p29l krb5kdc[14371](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:33:52 dist-test-slave-p29l krb5kdc[14371](info): set up 2 sockets
Sep 29 20:33:52 dist-test-slave-p29l krb5kdc[14371](info): commencing operation
krb5kdc: starting...
W20260929 20:33:53.148630 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.010s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:33:53 dist-test-slave-p29l krb5kdc[14371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:33:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:33:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.244.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
I20260929 20:33:53.420332 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:36473
--webserver_interface=127.28.244.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:38253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:36473
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:33:53.775279 14384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:33:53.775616 14384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:33:53.775779 14384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:33:53.783751 14384 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:33:53.783927 14384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:33:53.784031 14384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:33:53.784132 14384 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:33:53.784214 14384 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:33:53.795027 14384 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:38253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:36473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:36473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:33:53.796703 14384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:33:53.798009 14384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:33:53.810477 14390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:33:53.810448 14389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:33:53.810850 14392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:33:53.811375 14384 server_base.cc:1034] running on GCE node
I20260929 20:33:53.812368 14384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:33:53.814234 14384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:33:53.815470 14384 hybrid_clock.cc:648] HybridClock initialized: now 1790714033815410 us; error 46 us; skew 500 ppm
I20260929 20:33:53.815935 14384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:33:53 dist-test-slave-p29l krb5kdc[14371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714033, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:33:53.819435 14384 init.cc:377] Logged in from keytab as kudu/127.28.244.190@KRBTEST.COM (short username kudu)
I20260929 20:33:53.821228 14384 webserver.cc:460] Webserver started at http://127.28.244.190:39919/ using document root <none> and password file <none>
I20260929 20:33:53.821982 14384 fs_manager.cc:359] Metadata directory not provided
I20260929 20:33:53.822120 14384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:33:53.822450 14384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:33:53.825757 14384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "06d12dc5acff46548f0e3003d5ca9b5d"
format_stamp: "Formatted at 2026-09-29 20:33:53 on dist-test-slave-p29l"
server_key: "b60fdeddb5d74fe19a6017d610898acd"
server_key_iv: "9a0f2c0234b97bc4fa2beb950a1d69c0"
server_key_version: "encryptionkey@0"
I20260929 20:33:53.826596 14384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "06d12dc5acff46548f0e3003d5ca9b5d"
format_stamp: "Formatted at 2026-09-29 20:33:53 on dist-test-slave-p29l"
server_key: "b60fdeddb5d74fe19a6017d610898acd"
server_key_iv: "9a0f2c0234b97bc4fa2beb950a1d69c0"
server_key_version: "encryptionkey@0"
I20260929 20:33:53.832865 14384 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 20:33:53.836961 14399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:33:53.838487 14384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260929 20:33:53.838698 14384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "06d12dc5acff46548f0e3003d5ca9b5d"
format_stamp: "Formatted at 2026-09-29 20:33:53 on dist-test-slave-p29l"
server_key: "b60fdeddb5d74fe19a6017d610898acd"
server_key_iv: "9a0f2c0234b97bc4fa2beb950a1d69c0"
server_key_version: "encryptionkey@0"
I20260929 20:33:53.838937 14384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:33:53.856134 14384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:33:53.860987 14384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:33:53.861311 14384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:33:53.881057 14384 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:36473
I20260929 20:33:53.881076 14452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:36473 every 8 connection(s)
I20260929 20:33:53.882556 14384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:33:53.886586 14453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:33:53.888235 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 14384
I20260929 20:33:53.888475 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:33:53.889318 29650 external_mini_cluster.cc:1428] Setting key 9c25f4f79ffd65cbb04a3dfc3aa3a0e7
I20260929 20:33:53.902269 14453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06d12dc5acff46548f0e3003d5ca9b5d: Bootstrap starting.
Sep 29 20:33:53 dist-test-slave-p29l krb5kdc[14371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:33:53.906651 14453 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06d12dc5acff46548f0e3003d5ca9b5d: Neither blocks nor log segments found. Creating new log.
I20260929 20:33:53.907752 14453 log.cc:824] T 00000000000000000000000000000000 P 06d12dc5acff46548f0e3003d5ca9b5d: Log is configured to *not* fsync() on all Append() calls
I20260929 20:33:53.911346 14453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06d12dc5acff46548f0e3003d5ca9b5d: No bootstrap required, opened a new log
I20260929 20:33:53.915468 14456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:53.891570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53176 (local address 127.28.244.190:36473)
0929 20:33:53.892552 (+ 982us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:53.892565 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:53.892613 (+ 48us) server_negotiation.cc:407] Connection header received
0929 20:33:53.894347 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:53.894375 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:53.894838 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:53.895191 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:33:53.897321 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:53.898832 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:53.900855 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:53.901641 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:53.901801 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:53.907290 (+ 5489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:33:53.907299 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:33:53.907315 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:33:53.907365 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:33:53.910875 (+ 3510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:53.911772 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:53.911780 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:53.911786 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:53.912033 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:53.912452 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:53.912456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:53.912458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:53.913040 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:33:53.913382 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:53.913885 (+ 503us) server_negotiation.cc:299] Negotiation successful
0929 20:33:53.914367 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":745,"thread_start_us":504,"threads_started":1}
I20260929 20:33:53.920802 14453 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06d12dc5acff46548f0e3003d5ca9b5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d12dc5acff46548f0e3003d5ca9b5d" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 36473 } }
I20260929 20:33:53.921380 14453 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06d12dc5acff46548f0e3003d5ca9b5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:33:53.921494 14453 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06d12dc5acff46548f0e3003d5ca9b5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06d12dc5acff46548f0e3003d5ca9b5d, State: Initialized, Role: FOLLOWER
I20260929 20:33:53.922062 14453 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06d12dc5acff46548f0e3003d5ca9b5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d12dc5acff46548f0e3003d5ca9b5d" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 36473 } }
I20260929 20:33:53.922278 14453 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06d12dc5acff46548f0e3003d5ca9b5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:33:53.922377 14453 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06d12dc5acff46548f0e3003d5ca9b5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:33:53.922506 14453 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06d12dc5acff46548f0e3003d5ca9b5d [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:33:53.924820 14453 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06d12dc5acff46548f0e3003d5ca9b5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d12dc5acff46548f0e3003d5ca9b5d" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 36473 } }
I20260929 20:33:53.925323 14453 leader_election.cc:302] T 00000000000000000000000000000000 P 06d12dc5acff46548f0e3003d5ca9b5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06d12dc5acff46548f0e3003d5ca9b5d; no voters:
I20260929 20:33:53.925921 14453 leader_election.cc:288] T 00000000000000000000000000000000 P 06d12dc5acff46548f0e3003d5ca9b5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:33:53.926086 14458 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06d12dc5acff46548f0e3003d5ca9b5d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:33:53.926846 14458 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06d12dc5acff46548f0e3003d5ca9b5d [term 1 LEADER]: Becoming Leader. State: Replica: 06d12dc5acff46548f0e3003d5ca9b5d, State: Running, Role: LEADER
I20260929 20:33:53.927385 14458 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06d12dc5acff46548f0e3003d5ca9b5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d12dc5acff46548f0e3003d5ca9b5d" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 36473 } }
I20260929 20:33:53.928311 14453 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06d12dc5acff46548f0e3003d5ca9b5d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:33:53.930678 14459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06d12dc5acff46548f0e3003d5ca9b5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06d12dc5acff46548f0e3003d5ca9b5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d12dc5acff46548f0e3003d5ca9b5d" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 36473 } } }
I20260929 20:33:53.930663 14460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06d12dc5acff46548f0e3003d5ca9b5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06d12dc5acff46548f0e3003d5ca9b5d. Latest consensus state: current_term: 1 leader_uuid: "06d12dc5acff46548f0e3003d5ca9b5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d12dc5acff46548f0e3003d5ca9b5d" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 36473 } } }
I20260929 20:33:53.931923 14459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06d12dc5acff46548f0e3003d5ca9b5d [sys.catalog]: This master's current role is: LEADER
I20260929 20:33:53.932109 14460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06d12dc5acff46548f0e3003d5ca9b5d [sys.catalog]: This master's current role is: LEADER
I20260929 20:33:53.937088 14466 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:33:53.943215 14466 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:33:53.952803 14466 catalog_manager.cc:1347] Generated new cluster ID: b74638e4a3ea4530a419dcc594912ca4
I20260929 20:33:53.952945 14466 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:33:53.983423 14466 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:33:53.985517 14466 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:33:53.998554 14466 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06d12dc5acff46548f0e3003d5ca9b5d: Generated new TSK 0
I20260929 20:33:53.999850 14466 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.244.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab.
Entry for principal kudu/127.28.244.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab.
I20260929 20:33:54.068468 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.129:0
--local_ip_for_outbound_sockets=127.28.244.129
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:36473
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:38253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:33:54.427285 14480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:33:54.427629 14480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:33:54.427793 14480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:33:54.436025 14480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:33:54.436149 14480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:33:54.436321 14480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.129
I20260929 20:33:54.446830 14480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:38253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:36473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.129
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:33:54.448360 14480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:33:54.449767 14480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:33:54.463940 14485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:33:54.463918 14486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:33:54.463940 14488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:33:54.464833 14480 server_base.cc:1034] running on GCE node
I20260929 20:33:54.465584 14480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:33:54.466616 14480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:33:54.467922 14480 hybrid_clock.cc:648] HybridClock initialized: now 1790714034467860 us; error 43 us; skew 500 ppm
I20260929 20:33:54.468308 14480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:33:54 dist-test-slave-p29l krb5kdc[14371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714034, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:33:54.471975 14480 init.cc:377] Logged in from keytab as kudu/127.28.244.129@KRBTEST.COM (short username kudu)
I20260929 20:33:54.473862 14480 webserver.cc:460] Webserver started at http://127.28.244.129:33897/ using document root <none> and password file <none>
I20260929 20:33:54.474663 14480 fs_manager.cc:359] Metadata directory not provided
I20260929 20:33:54.474829 14480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:33:54.475193 14480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:33:54.478770 14480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-0/data/instance:
uuid: "8be4ea8f027d452290ed47e52d7a6ae0"
format_stamp: "Formatted at 2026-09-29 20:33:54 on dist-test-slave-p29l"
server_key: "76fe95372cb6a1f0822a49eb37102a81"
server_key_iv: "42aa0d4d6a1e078113c67a77d428b23f"
server_key_version: "encryptionkey@0"
I20260929 20:33:54.479714 14480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance:
uuid: "8be4ea8f027d452290ed47e52d7a6ae0"
format_stamp: "Formatted at 2026-09-29 20:33:54 on dist-test-slave-p29l"
server_key: "76fe95372cb6a1f0822a49eb37102a81"
server_key_iv: "42aa0d4d6a1e078113c67a77d428b23f"
server_key_version: "encryptionkey@0"
I20260929 20:33:54.485788 14480 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260929 20:33:54.489861 14495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:33:54.491375 14480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:33:54.491573 14480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-0/wal
uuid: "8be4ea8f027d452290ed47e52d7a6ae0"
format_stamp: "Formatted at 2026-09-29 20:33:54 on dist-test-slave-p29l"
server_key: "76fe95372cb6a1f0822a49eb37102a81"
server_key_iv: "42aa0d4d6a1e078113c67a77d428b23f"
server_key_version: "encryptionkey@0"
I20260929 20:33:54.491792 14480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:33:54.519498 14480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:33:54.524137 14480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:33:54.524528 14480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:33:54.525599 14480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:33:54.527032 14480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:33:54.527143 14480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:33:54.527295 14480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:33:54.527364 14480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:33:54.562691 14480 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.129:39649
I20260929 20:33:54.562806 14609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.129:39649 every 8 connection(s)
I20260929 20:33:54.564114 14480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
I20260929 20:33:54.565594 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 14480
I20260929 20:33:54.565798 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance
I20260929 20:33:54.566588 29650 external_mini_cluster.cc:1428] Setting key 5cd4bf1d069c8bdaa80063c11d3a00ab
Sep 29 20:33:54 dist-test-slave-p29l krb5kdc[14371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714034, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:33:54.588198 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:54.568084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:60531 (local address 127.28.244.190:36473)
0929 20:33:54.570944 (+ 2860us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:54.570948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:54.570963 (+ 15us) server_negotiation.cc:407] Connection header received
0929 20:33:54.571016 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:54.571019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:54.571078 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:54.571201 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:33:54.572902 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:54.573634 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:54.575207 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:54.575623 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:54.575757 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:54.579626 (+ 3869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:33:54.579637 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:33:54.579647 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:33:54.579700 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:33:54.583985 (+ 4285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:54.585171 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:54.585177 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:54.585181 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:54.585381 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:54.586043 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:54.586051 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:54.586054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:54.586447 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:33:54.586607 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:54.587507 (+ 900us) server_negotiation.cc:299] Negotiation successful
0929 20:33:54.587786 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2751,"thread_start_us":2660,"threads_started":1}
I20260929 20:33:54.589380 14612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:54.567621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:36473 (local address 127.28.244.129:60531)
0929 20:33:54.568568 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:54.568617 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 20:33:54.570384 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:33:54.571485 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:33:54.571494 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:33:54.572115 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:33:54.572693 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:54.572728 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:54.573816 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:54.573820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:54.575056 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:54.575068 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:54.575897 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:54.575904 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:54.576134 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:54.576908 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:33:54.576935 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:33:54.579365 (+ 2430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:33:54.584189 (+ 4824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:33:54.584201 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:33:54.584221 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:33:54.584991 (+ 770us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:33:54.585529 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:33:54.585536 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:33:54.585541 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:33:54.585881 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:33:54.586577 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:33:54.586586 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:33:54.587093 (+ 507us) client_negotiation.cc:769] Sending connection context
0929 20:33:54.587384 (+ 291us) client_negotiation.cc:240] Negotiation successful
0929 20:33:54.587996 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":435,"threads_started":1}
I20260929 20:33:54.591374 14610 heartbeater.cc:347] Connected to a master server at 127.28.244.190:36473
I20260929 20:33:54.591717 14610 heartbeater.cc:464] Registering TS with master...
I20260929 20:33:54.592587 14610 heartbeater.cc:511] Master 127.28.244.190:36473 requested a full tablet report, sending...
I20260929 20:33:54.595036 14416 ts_manager.cc:194] Registered new tserver with Master: 8be4ea8f027d452290ed47e52d7a6ae0 (127.28.244.129:39649)
WARNING: no policy specified for kudu/127.28.244.130@KRBTEST.COM; defaulting to no policy
I20260929 20:33:54.596691 14416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.129@KRBTEST.COM'} at 127.28.244.129:60531
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab.
Entry for principal kudu/127.28.244.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab.
I20260929 20:33:54.631623 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.130:0
--local_ip_for_outbound_sockets=127.28.244.130
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:36473
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:38253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:33:54.995169 14617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:33:54.995515 14617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:33:54.995673 14617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:33:55.003651 14617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:33:55.003850 14617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:33:55.004057 14617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.130
I20260929 20:33:55.014586 14617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:38253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:36473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.130
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:33:55.016077 14617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:33:55.017434 14617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:33:55.031351 14622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:33:55.031432 14623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:33:55.031385 14625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:33:55.032841 14617 server_base.cc:1034] running on GCE node
I20260929 20:33:55.033509 14617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:33:55.034595 14617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:33:55.035861 14617 hybrid_clock.cc:648] HybridClock initialized: now 1790714035035762 us; error 54 us; skew 500 ppm
I20260929 20:33:55.036204 14617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:33:55 dist-test-slave-p29l krb5kdc[14371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714035, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:33:55.039471 14617 init.cc:377] Logged in from keytab as kudu/127.28.244.130@KRBTEST.COM (short username kudu)
I20260929 20:33:55.041157 14617 webserver.cc:460] Webserver started at http://127.28.244.130:42463/ using document root <none> and password file <none>
I20260929 20:33:55.041877 14617 fs_manager.cc:359] Metadata directory not provided
I20260929 20:33:55.042018 14617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:33:55.042336 14617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:33:55.045603 14617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-1/data/instance:
uuid: "784295b44bb04262a2d6152512ad3c43"
format_stamp: "Formatted at 2026-09-29 20:33:55 on dist-test-slave-p29l"
server_key: "41b3b77a28c736d7e71038736330d8ff"
server_key_iv: "b4157d75a569aab14820ab8fdbd1d180"
server_key_version: "encryptionkey@0"
I20260929 20:33:55.046424 14617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance:
uuid: "784295b44bb04262a2d6152512ad3c43"
format_stamp: "Formatted at 2026-09-29 20:33:55 on dist-test-slave-p29l"
server_key: "41b3b77a28c736d7e71038736330d8ff"
server_key_iv: "b4157d75a569aab14820ab8fdbd1d180"
server_key_version: "encryptionkey@0"
I20260929 20:33:55.052333 14617 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260929 20:33:55.056248 14632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:33:55.057789 14617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:33:55.057996 14617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-1/wal
uuid: "784295b44bb04262a2d6152512ad3c43"
format_stamp: "Formatted at 2026-09-29 20:33:55 on dist-test-slave-p29l"
server_key: "41b3b77a28c736d7e71038736330d8ff"
server_key_iv: "b4157d75a569aab14820ab8fdbd1d180"
server_key_version: "encryptionkey@0"
I20260929 20:33:55.058226 14617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:33:55.073827 14617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:33:55.078634 14617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:33:55.079085 14617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:33:55.080265 14617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:33:55.081746 14617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:33:55.081947 14617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:33:55.082115 14617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:33:55.082206 14617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:33:55.118467 14617 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.130:39717
I20260929 20:33:55.118510 14745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.130:39717 every 8 connection(s)
I20260929 20:33:55.119971 14617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
I20260929 20:33:55.128306 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 14617
I20260929 20:33:55.128536 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance
I20260929 20:33:55.129091 29650 external_mini_cluster.cc:1428] Setting key 6b999d5002ed1cfdcd3a1259491af2d5
Sep 29 20:33:55 dist-test-slave-p29l krb5kdc[14371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714035, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:33:55.144678 14748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:55.122815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:52267 (local address 127.28.244.190:36473)
0929 20:33:55.123336 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:55.123341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:55.124376 (+ 1035us) server_negotiation.cc:407] Connection header received
0929 20:33:55.126674 (+ 2298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:55.126680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:55.126785 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:55.126910 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:33:55.129009 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:55.130138 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:55.132375 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:55.132836 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:55.132943 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:55.137725 (+ 4782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:33:55.137731 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:33:55.137734 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:33:55.137774 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:33:55.140773 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:55.141665 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:55.141669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:55.141672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:55.141812 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:55.142313 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:55.142320 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:55.142325 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:55.142686 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:33:55.142861 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:55.143767 (+ 906us) server_negotiation.cc:299] Negotiation successful
0929 20:33:55.144173 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":283,"threads_started":1}
I20260929 20:33:55.145419 14749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:55.123285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:36473 (local address 127.28.244.130:52267)
0929 20:33:55.124179 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:55.124228 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 20:33:55.126342 (+ 2114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:33:55.127171 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:33:55.127181 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:33:55.127734 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:33:55.128758 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:55.128786 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:55.130349 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:55.130355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:55.132171 (+ 1816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:55.132190 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:55.133101 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:55.133111 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:55.133499 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:55.134357 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:33:55.134395 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:33:55.137402 (+ 3007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:33:55.140968 (+ 3566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:33:55.140979 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:33:55.141000 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:33:55.141488 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:33:55.141943 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:33:55.141949 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:33:55.141952 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:33:55.142176 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:33:55.142854 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:33:55.142864 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:33:55.143280 (+ 416us) client_negotiation.cc:769] Sending connection context
0929 20:33:55.143620 (+ 340us) client_negotiation.cc:240] Negotiation successful
0929 20:33:55.144168 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":303,"threads_started":1}
I20260929 20:33:55.147213 14746 heartbeater.cc:347] Connected to a master server at 127.28.244.190:36473
I20260929 20:33:55.147588 14746 heartbeater.cc:464] Registering TS with master...
I20260929 20:33:55.148646 14746 heartbeater.cc:511] Master 127.28.244.190:36473 requested a full tablet report, sending...
I20260929 20:33:55.150427 14416 ts_manager.cc:194] Registered new tserver with Master: 784295b44bb04262a2d6152512ad3c43 (127.28.244.130:39717)
I20260929 20:33:55.151340 14416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.130@KRBTEST.COM'} at 127.28.244.130:52267
WARNING: no policy specified for kudu/127.28.244.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab.
Entry for principal kudu/127.28.244.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab.
I20260929 20:33:55.192293 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.131:0
--local_ip_for_outbound_sockets=127.28.244.131
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:36473
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:38253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:33:55.550406 14752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:33:55.550729 14752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:33:55.550827 14752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:33:55.558481 14752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:33:55.558615 14752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:33:55.558732 14752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.131
I20260929 20:33:55.571556 14752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:38253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:36473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.131
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:33:55.572942 14752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:33:55.574271 14752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:33:55.588838 14759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:33:55.588886 14761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:33:55.589289 14758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:33:55.591423 14752 server_base.cc:1034] running on GCE node
I20260929 20:33:55.592243 14752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:33:55.593286 14752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:33:55.594554 14752 hybrid_clock.cc:648] HybridClock initialized: now 1790714035594477 us; error 60 us; skew 500 ppm
I20260929 20:33:55.594925 14752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:33:55 dist-test-slave-p29l krb5kdc[14371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714035, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:33:55.598407 14752 init.cc:377] Logged in from keytab as kudu/127.28.244.131@KRBTEST.COM (short username kudu)
I20260929 20:33:55.599939 14752 webserver.cc:460] Webserver started at http://127.28.244.131:46779/ using document root <none> and password file <none>
I20260929 20:33:55.600212 14610 heartbeater.cc:503] Master 127.28.244.190:36473 was elected leader, sending a full tablet report...
I20260929 20:33:55.600824 14752 fs_manager.cc:359] Metadata directory not provided
I20260929 20:33:55.600977 14752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:33:55.601307 14752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:33:55.604437 14752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-2/data/instance:
uuid: "e7b2f19d10334c528ecde9b8263c4ab0"
format_stamp: "Formatted at 2026-09-29 20:33:55 on dist-test-slave-p29l"
server_key: "265ba22eb6af9cbd3c8261105a7a3d04"
server_key_iv: "45eadd113a020c8531486719227acfe9"
server_key_version: "encryptionkey@0"
I20260929 20:33:55.605211 14752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance:
uuid: "e7b2f19d10334c528ecde9b8263c4ab0"
format_stamp: "Formatted at 2026-09-29 20:33:55 on dist-test-slave-p29l"
server_key: "265ba22eb6af9cbd3c8261105a7a3d04"
server_key_iv: "45eadd113a020c8531486719227acfe9"
server_key_version: "encryptionkey@0"
I20260929 20:33:55.610847 14752 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:33:55.614595 14768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:33:55.616220 14752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:33:55.616426 14752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-2/wal
uuid: "e7b2f19d10334c528ecde9b8263c4ab0"
format_stamp: "Formatted at 2026-09-29 20:33:55 on dist-test-slave-p29l"
server_key: "265ba22eb6af9cbd3c8261105a7a3d04"
server_key_iv: "45eadd113a020c8531486719227acfe9"
server_key_version: "encryptionkey@0"
I20260929 20:33:55.616659 14752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:33:55.648640 14752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:33:55.653950 14752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:33:55.654249 14752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:33:55.655272 14752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:33:55.656778 14752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:33:55.656862 14752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:33:55.656955 14752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:33:55.657001 14752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:33:55.693176 14752 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.131:42291
I20260929 20:33:55.693264 14881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.131:42291 every 8 connection(s)
I20260929 20:33:55.694581 14752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
I20260929 20:33:55.699497 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 14752
I20260929 20:33:55.699692 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance
I20260929 20:33:55.700168 29650 external_mini_cluster.cc:1428] Setting key 0c7188049c85b69716a84b3a7050172e
Sep 29 20:33:55 dist-test-slave-p29l krb5kdc[14371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714035, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:33:55.716405 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:55.697376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:39243 (local address 127.28.244.190:36473)
0929 20:33:55.697893 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:55.697899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:55.698939 (+ 1040us) server_negotiation.cc:407] Connection header received
0929 20:33:55.701015 (+ 2076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:55.701020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:55.701090 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:55.701224 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:33:55.702996 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:55.703944 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:55.705602 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:55.706011 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:55.706110 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:55.709896 (+ 3786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:33:55.709901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:33:55.709905 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:33:55.709940 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:33:55.712816 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:55.713638 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:55.713644 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:55.713648 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:55.713782 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:55.714297 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:55.714310 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:55.714312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:55.714732 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:33:55.714896 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:55.715750 (+ 854us) server_negotiation.cc:299] Negotiation successful
0929 20:33:55.716006 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":230,"threads_started":1}
I20260929 20:33:55.717370 14885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:55.698014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:36473 (local address 127.28.244.131:39243)
0929 20:33:55.698767 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:55.698809 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 20:33:55.700704 (+ 1895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:33:55.701517 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:33:55.701536 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:33:55.702175 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:33:55.702823 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:55.702842 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:55.704106 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:55.704111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:55.705447 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:55.705462 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:55.706230 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:55.706236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:55.706446 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:55.707141 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:33:55.707169 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:33:55.709653 (+ 2484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:33:55.713006 (+ 3353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:33:55.713016 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:33:55.713042 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:33:55.713497 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:33:55.713893 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:33:55.713899 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:33:55.713902 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:33:55.714165 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:33:55.714863 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:33:55.714871 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:33:55.715300 (+ 429us) client_negotiation.cc:769] Sending connection context
0929 20:33:55.715650 (+ 350us) client_negotiation.cc:240] Negotiation successful
0929 20:33:55.716244 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":304,"threads_started":1}
I20260929 20:33:55.718693 14882 heartbeater.cc:347] Connected to a master server at 127.28.244.190:36473
I20260929 20:33:55.719058 14882 heartbeater.cc:464] Registering TS with master...
I20260929 20:33:55.719935 14882 heartbeater.cc:511] Master 127.28.244.190:36473 requested a full tablet report, sending...
I20260929 20:33:55.721540 14416 ts_manager.cc:194] Registered new tserver with Master: e7b2f19d10334c528ecde9b8263c4ab0 (127.28.244.131:42291)
I20260929 20:33:55.722535 14416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.131@KRBTEST.COM'} at 127.28.244.131:39243
I20260929 20:33:55.728811 29650 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:33:55.749240 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:55.733855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53182 (local address 127.28.244.190:36473)
0929 20:33:55.734165 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:55.734174 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:55.734487 (+ 313us) server_negotiation.cc:407] Connection header received
0929 20:33:55.734727 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:55.734732 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:55.734801 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:55.734912 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:33:55.736622 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:55.737492 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:55.739319 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:55.739799 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:55.740028 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:55.742862 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:33:55.742868 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:33:55.742871 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:33:55.742915 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:33:55.745910 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:55.746817 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:55.746833 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:55.746836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:55.747003 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:55.747498 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:55.747503 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:55.747506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:55.747992 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:33:55.748265 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:55.748653 (+ 388us) server_negotiation.cc:299] Negotiation successful
0929 20:33:55.748908 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260929 20:33:55.753933 14416 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:33:55.761137 14416 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:33:55.796675 14898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:55.775977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41600 (local address 127.28.244.129:39649)
0929 20:33:55.785302 (+ 9325us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:55.785316 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:55.787245 (+ 1929us) server_negotiation.cc:407] Connection header received
0929 20:33:55.787362 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:55.787372 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:55.787631 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:55.788933 (+ 1302us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:33:55.789174 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:55.790829 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:55.793596 (+ 2767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:55.795196 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:55.795422 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:55.795698 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:55.796079 (+ 381us) server_negotiation.cc:299] Negotiation successful
0929 20:33:55.796280 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9064,"thread_start_us":8859,"threads_started":1}
I20260929 20:33:55.796886 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:55.774753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41948 (local address 127.28.244.130:39717)
0929 20:33:55.786485 (+ 11732us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:55.786495 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:55.786528 (+ 33us) server_negotiation.cc:407] Connection header received
0929 20:33:55.787046 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:55.787056 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:55.787377 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:55.787571 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:33:55.789278 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:55.790838 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:55.793809 (+ 2971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:55.795394 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:55.795595 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:55.795741 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:55.796329 (+ 588us) server_negotiation.cc:299] Negotiation successful
0929 20:33:55.796516 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11559,"thread_start_us":11446,"threads_started":1}
I20260929 20:33:55.796973 14896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:55.775229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:39649 (local address 127.0.0.1:41600)
0929 20:33:55.786951 (+ 11722us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:55.786979 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 20:33:55.787105 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:33:55.787934 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:33:55.787954 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:33:55.788031 (+ 77us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:33:55.788678 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:55.788691 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:55.791274 (+ 2583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:55.791279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:55.793422 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:55.793435 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:55.795449 (+ 2014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:55.795456 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:55.795739 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:55.795824 (+ 85us) client_negotiation.cc:769] Sending connection context
0929 20:33:55.795977 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 20:33:55.796271 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11565,"mutex_wait_us":53,"spinlock_wait_cycles":10880,"thread_start_us":10103,"threads_started":1}
I20260929 20:33:55.797514 14899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:55.787301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:42291 (local address 127.0.0.1:38042)
0929 20:33:55.788975 (+ 1674us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:55.788999 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:33:55.789149 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:33:55.790451 (+ 1302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:33:55.790457 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:33:55.790484 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:33:55.790814 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:55.790823 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:55.792865 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:55.792870 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:55.794644 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:55.794656 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:55.796634 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:55.796640 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:55.796814 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:55.796853 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 20:33:55.796960 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 20:33:55.797125 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1488,"mutex_wait_us":52,"thread_start_us":1276,"threads_started":1}
I20260929 20:33:55.798107 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:55.774733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:39717 (local address 127.0.0.1:41948)
0929 20:33:55.785750 (+ 11017us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:55.785814 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 20:33:55.786724 (+ 910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:33:55.787930 (+ 1206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:33:55.787942 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:33:55.788015 (+ 73us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:33:55.789059 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:55.789073 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:55.791009 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:55.791015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:55.793620 (+ 2605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:55.793635 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:55.795887 (+ 2252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:55.795894 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:55.796116 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:55.796148 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 20:33:55.797564 (+ 1416us) client_negotiation.cc:240] Negotiation successful
0929 20:33:55.797721 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10751,"mutex_wait_us":531,"spinlock_wait_cycles":3968,"thread_start_us":11903,"threads_started":1}
I20260929 20:33:55.798702 14900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:55.787452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38042 (local address 127.28.244.131:42291)
0929 20:33:55.789753 (+ 2301us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:55.789759 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:55.789787 (+ 28us) server_negotiation.cc:407] Connection header received
0929 20:33:55.789860 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:55.789865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:55.790106 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:55.790301 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:33:55.790990 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:55.792691 (+ 1701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:55.794801 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:55.796129 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:55.796312 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:55.796490 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:55.797817 (+ 1327us) server_negotiation.cc:299] Negotiation successful
0929 20:33:55.797982 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2143,"thread_start_us":1970,"threads_started":1}
I20260929 20:33:55.801672 14680 tablet_service.cc:1467] Processing CreateTablet for tablet a433e6521faa4f76ab831b18849289e3 (DEFAULT_TABLE table=test-table [id=4a5b2cf833914c85acf1bccb7863bba0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:33:55.801702 14544 tablet_service.cc:1467] Processing CreateTablet for tablet a433e6521faa4f76ab831b18849289e3 (DEFAULT_TABLE table=test-table [id=4a5b2cf833914c85acf1bccb7863bba0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:33:55.802194 14816 tablet_service.cc:1467] Processing CreateTablet for tablet a433e6521faa4f76ab831b18849289e3 (DEFAULT_TABLE table=test-table [id=4a5b2cf833914c85acf1bccb7863bba0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:33:55.803103 14680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a433e6521faa4f76ab831b18849289e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:33:55.803279 14544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a433e6521faa4f76ab831b18849289e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:33:55.803565 14816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a433e6521faa4f76ab831b18849289e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:33:55.816133 14901 tablet_bootstrap.cc:492] T a433e6521faa4f76ab831b18849289e3 P 784295b44bb04262a2d6152512ad3c43: Bootstrap starting.
I20260929 20:33:55.818542 14902 tablet_bootstrap.cc:492] T a433e6521faa4f76ab831b18849289e3 P 8be4ea8f027d452290ed47e52d7a6ae0: Bootstrap starting.
I20260929 20:33:55.819442 14904 tablet_bootstrap.cc:492] T a433e6521faa4f76ab831b18849289e3 P e7b2f19d10334c528ecde9b8263c4ab0: Bootstrap starting.
I20260929 20:33:55.820067 14901 tablet_bootstrap.cc:654] T a433e6521faa4f76ab831b18849289e3 P 784295b44bb04262a2d6152512ad3c43: Neither blocks nor log segments found. Creating new log.
I20260929 20:33:55.821507 14901 log.cc:824] T a433e6521faa4f76ab831b18849289e3 P 784295b44bb04262a2d6152512ad3c43: Log is configured to *not* fsync() on all Append() calls
I20260929 20:33:55.823607 14904 tablet_bootstrap.cc:654] T a433e6521faa4f76ab831b18849289e3 P e7b2f19d10334c528ecde9b8263c4ab0: Neither blocks nor log segments found. Creating new log.
I20260929 20:33:55.824577 14901 tablet_bootstrap.cc:492] T a433e6521faa4f76ab831b18849289e3 P 784295b44bb04262a2d6152512ad3c43: No bootstrap required, opened a new log
I20260929 20:33:55.824872 14901 ts_tablet_manager.cc:1397] T a433e6521faa4f76ab831b18849289e3 P 784295b44bb04262a2d6152512ad3c43: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260929 20:33:55.825014 14904 log.cc:824] T a433e6521faa4f76ab831b18849289e3 P e7b2f19d10334c528ecde9b8263c4ab0: Log is configured to *not* fsync() on all Append() calls
I20260929 20:33:55.827490 14902 tablet_bootstrap.cc:654] T a433e6521faa4f76ab831b18849289e3 P 8be4ea8f027d452290ed47e52d7a6ae0: Neither blocks nor log segments found. Creating new log.
I20260929 20:33:55.828824 14902 log.cc:824] T a433e6521faa4f76ab831b18849289e3 P 8be4ea8f027d452290ed47e52d7a6ae0: Log is configured to *not* fsync() on all Append() calls
I20260929 20:33:55.831353 14904 tablet_bootstrap.cc:492] T a433e6521faa4f76ab831b18849289e3 P e7b2f19d10334c528ecde9b8263c4ab0: No bootstrap required, opened a new log
I20260929 20:33:55.831749 14904 ts_tablet_manager.cc:1397] T a433e6521faa4f76ab831b18849289e3 P e7b2f19d10334c528ecde9b8263c4ab0: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260929 20:33:55.833637 14901 raft_consensus.cc:348] T a433e6521faa4f76ab831b18849289e3 P 784295b44bb04262a2d6152512ad3c43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "784295b44bb04262a2d6152512ad3c43" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 39717 } } peers { permanent_uuid: "e7b2f19d10334c528ecde9b8263c4ab0" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 42291 } } peers { permanent_uuid: "8be4ea8f027d452290ed47e52d7a6ae0" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 39649 } }
I20260929 20:33:55.834087 14901 raft_consensus.cc:374] T a433e6521faa4f76ab831b18849289e3 P 784295b44bb04262a2d6152512ad3c43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:33:55.834192 14901 raft_consensus.cc:729] T a433e6521faa4f76ab831b18849289e3 P 784295b44bb04262a2d6152512ad3c43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 784295b44bb04262a2d6152512ad3c43, State: Initialized, Role: FOLLOWER
I20260929 20:33:55.834933 14901 consensus_queue.cc:260] T a433e6521faa4f76ab831b18849289e3 P 784295b44bb04262a2d6152512ad3c43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "784295b44bb04262a2d6152512ad3c43" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 39717 } } peers { permanent_uuid: "e7b2f19d10334c528ecde9b8263c4ab0" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 42291 } } peers { permanent_uuid: "8be4ea8f027d452290ed47e52d7a6ae0" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 39649 } }
I20260929 20:33:55.836592 14746 heartbeater.cc:503] Master 127.28.244.190:36473 was elected leader, sending a full tablet report...
I20260929 20:33:55.837467 14902 tablet_bootstrap.cc:492] T a433e6521faa4f76ab831b18849289e3 P 8be4ea8f027d452290ed47e52d7a6ae0: No bootstrap required, opened a new log
I20260929 20:33:55.837777 14902 ts_tablet_manager.cc:1397] T a433e6521faa4f76ab831b18849289e3 P 8be4ea8f027d452290ed47e52d7a6ae0: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.000s
I20260929 20:33:55.838809 14901 ts_tablet_manager.cc:1428] T a433e6521faa4f76ab831b18849289e3 P 784295b44bb04262a2d6152512ad3c43: Time spent starting tablet: real 0.014s user 0.010s sys 0.002s
I20260929 20:33:55.842669 14904 raft_consensus.cc:348] T a433e6521faa4f76ab831b18849289e3 P e7b2f19d10334c528ecde9b8263c4ab0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "784295b44bb04262a2d6152512ad3c43" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 39717 } } peers { permanent_uuid: "e7b2f19d10334c528ecde9b8263c4ab0" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 42291 } } peers { permanent_uuid: "8be4ea8f027d452290ed47e52d7a6ae0" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 39649 } }
I20260929 20:33:55.843202 14904 raft_consensus.cc:374] T a433e6521faa4f76ab831b18849289e3 P e7b2f19d10334c528ecde9b8263c4ab0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:33:55.843327 14904 raft_consensus.cc:729] T a433e6521faa4f76ab831b18849289e3 P e7b2f19d10334c528ecde9b8263c4ab0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7b2f19d10334c528ecde9b8263c4ab0, State: Initialized, Role: FOLLOWER
I20260929 20:33:55.844094 14904 consensus_queue.cc:260] T a433e6521faa4f76ab831b18849289e3 P e7b2f19d10334c528ecde9b8263c4ab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "784295b44bb04262a2d6152512ad3c43" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 39717 } } peers { permanent_uuid: "e7b2f19d10334c528ecde9b8263c4ab0" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 42291 } } peers { permanent_uuid: "8be4ea8f027d452290ed47e52d7a6ae0" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 39649 } }
I20260929 20:33:55.845271 14882 heartbeater.cc:503] Master 127.28.244.190:36473 was elected leader, sending a full tablet report...
I20260929 20:33:55.845842 14904 ts_tablet_manager.cc:1428] T a433e6521faa4f76ab831b18849289e3 P e7b2f19d10334c528ecde9b8263c4ab0: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260929 20:33:55.848528 14902 raft_consensus.cc:348] T a433e6521faa4f76ab831b18849289e3 P 8be4ea8f027d452290ed47e52d7a6ae0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "784295b44bb04262a2d6152512ad3c43" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 39717 } } peers { permanent_uuid: "e7b2f19d10334c528ecde9b8263c4ab0" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 42291 } } peers { permanent_uuid: "8be4ea8f027d452290ed47e52d7a6ae0" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 39649 } }
I20260929 20:33:55.849025 14902 raft_consensus.cc:374] T a433e6521faa4f76ab831b18849289e3 P 8be4ea8f027d452290ed47e52d7a6ae0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:33:55.849129 14902 raft_consensus.cc:729] T a433e6521faa4f76ab831b18849289e3 P 8be4ea8f027d452290ed47e52d7a6ae0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8be4ea8f027d452290ed47e52d7a6ae0, State: Initialized, Role: FOLLOWER
I20260929 20:33:55.849772 14902 consensus_queue.cc:260] T a433e6521faa4f76ab831b18849289e3 P 8be4ea8f027d452290ed47e52d7a6ae0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "784295b44bb04262a2d6152512ad3c43" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 39717 } } peers { permanent_uuid: "e7b2f19d10334c528ecde9b8263c4ab0" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 42291 } } peers { permanent_uuid: "8be4ea8f027d452290ed47e52d7a6ae0" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 39649 } }
I20260929 20:33:55.851311 14902 ts_tablet_manager.cc:1428] T a433e6521faa4f76ab831b18849289e3 P 8be4ea8f027d452290ed47e52d7a6ae0: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260929 20:33:55.872191 14747 tablet.cc:2369] T a433e6521faa4f76ab831b18849289e3 P 784295b44bb04262a2d6152512ad3c43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:33:55.918393 14910 raft_consensus.cc:482] T a433e6521faa4f76ab831b18849289e3 P 8be4ea8f027d452290ed47e52d7a6ae0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:33:55.918800 14910 raft_consensus.cc:504] T a433e6521faa4f76ab831b18849289e3 P 8be4ea8f027d452290ed47e52d7a6ae0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "784295b44bb04262a2d6152512ad3c43" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 39717 } } peers { permanent_uuid: "e7b2f19d10334c528ecde9b8263c4ab0" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 42291 } } peers { permanent_uuid: "8be4ea8f027d452290ed47e52d7a6ae0" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 39649 } }
I20260929 20:33:55.920688 14910 leader_election.cc:288] T a433e6521faa4f76ab831b18849289e3 P 8be4ea8f027d452290ed47e52d7a6ae0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 784295b44bb04262a2d6152512ad3c43 (127.28.244.130:39717), e7b2f19d10334c528ecde9b8263c4ab0 (127.28.244.131:42291)
I20260929 20:33:55.927311 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:55.921158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:44651 (local address 127.28.244.130:39717)
0929 20:33:55.921403 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:55.921410 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:55.921920 (+ 510us) server_negotiation.cc:407] Connection header received
0929 20:33:55.922197 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:55.922205 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:55.922304 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:55.922412 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:33:55.923002 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:55.923734 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:55.925524 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:55.926336 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:55.926521 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:55.926606 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:55.926884 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 20:33:55.927023 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:33:55.927704 14911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:55.921098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:39717 (local address 127.28.244.129:44651)
0929 20:33:55.921804 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:55.921820 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:33:55.922049 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:33:55.922467 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:33:55.922472 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:33:55.922512 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:33:55.922849 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:55.922858 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:55.923894 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:55.923903 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:55.925345 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:55.925358 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:55.926544 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:55.926550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:55.926718 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:55.926743 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 20:33:55.926854 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 20:33:55.926996 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":406,"threads_started":1}
I20260929 20:33:55.927870 14900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:55.922055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:47077 (local address 127.28.244.131:42291)
0929 20:33:55.922297 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:55.922303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:55.922758 (+ 455us) server_negotiation.cc:407] Connection header received
0929 20:33:55.922966 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:55.922970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:55.923027 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:55.923126 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:33:55.923637 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:55.924399 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:55.926078 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:55.926867 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:55.927058 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:55.927137 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:55.927437 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 20:33:55.927572 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:33:55.927949 14912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:55.921970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:42291 (local address 127.28.244.129:47077)
0929 20:33:55.922649 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:55.922667 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:33:55.922818 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:33:55.923140 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:33:55.923144 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:33:55.923167 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:33:55.923492 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:55.923501 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:55.924546 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:55.924551 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:55.925929 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:55.925942 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:55.927094 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:55.927100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:55.927275 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:55.927299 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 20:33:55.927411 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 20:33:55.927542 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":331,"threads_started":1}
I20260929 20:33:55.928819 14836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a433e6521faa4f76ab831b18849289e3" candidate_uuid: "8be4ea8f027d452290ed47e52d7a6ae0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7b2f19d10334c528ecde9b8263c4ab0" is_pre_election: true
I20260929 20:33:55.928809 14700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a433e6521faa4f76ab831b18849289e3" candidate_uuid: "8be4ea8f027d452290ed47e52d7a6ae0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "784295b44bb04262a2d6152512ad3c43" is_pre_election: true
I20260929 20:33:55.929425 14700 raft_consensus.cc:2454] T a433e6521faa4f76ab831b18849289e3 P 784295b44bb04262a2d6152512ad3c43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8be4ea8f027d452290ed47e52d7a6ae0 in term 0.
I20260929 20:33:55.929410 14836 raft_consensus.cc:2454] T a433e6521faa4f76ab831b18849289e3 P e7b2f19d10334c528ecde9b8263c4ab0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8be4ea8f027d452290ed47e52d7a6ae0 in term 0.
I20260929 20:33:55.930146 14496 leader_election.cc:302] T a433e6521faa4f76ab831b18849289e3 P 8be4ea8f027d452290ed47e52d7a6ae0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 784295b44bb04262a2d6152512ad3c43, 8be4ea8f027d452290ed47e52d7a6ae0; no voters:
I20260929 20:33:55.930573 14910 raft_consensus.cc:2781] T a433e6521faa4f76ab831b18849289e3 P 8be4ea8f027d452290ed47e52d7a6ae0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:33:55.930732 14910 raft_consensus.cc:482] T a433e6521faa4f76ab831b18849289e3 P 8be4ea8f027d452290ed47e52d7a6ae0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:33:55.930842 14910 raft_consensus.cc:3037] T a433e6521faa4f76ab831b18849289e3 P 8be4ea8f027d452290ed47e52d7a6ae0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:33:55.933254 14910 raft_consensus.cc:504] T a433e6521faa4f76ab831b18849289e3 P 8be4ea8f027d452290ed47e52d7a6ae0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "784295b44bb04262a2d6152512ad3c43" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 39717 } } peers { permanent_uuid: "e7b2f19d10334c528ecde9b8263c4ab0" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 42291 } } peers { permanent_uuid: "8be4ea8f027d452290ed47e52d7a6ae0" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 39649 } }
I20260929 20:33:55.933883 14910 leader_election.cc:288] T a433e6521faa4f76ab831b18849289e3 P 8be4ea8f027d452290ed47e52d7a6ae0 [CANDIDATE]: Term 1 election: Requested vote from peers 784295b44bb04262a2d6152512ad3c43 (127.28.244.130:39717), e7b2f19d10334c528ecde9b8263c4ab0 (127.28.244.131:42291)
I20260929 20:33:55.934451 14700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a433e6521faa4f76ab831b18849289e3" candidate_uuid: "8be4ea8f027d452290ed47e52d7a6ae0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "784295b44bb04262a2d6152512ad3c43"
I20260929 20:33:55.934684 14700 raft_consensus.cc:3037] T a433e6521faa4f76ab831b18849289e3 P 784295b44bb04262a2d6152512ad3c43 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:33:55.934793 14836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a433e6521faa4f76ab831b18849289e3" candidate_uuid: "8be4ea8f027d452290ed47e52d7a6ae0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7b2f19d10334c528ecde9b8263c4ab0"
I20260929 20:33:55.935024 14836 raft_consensus.cc:3037] T a433e6521faa4f76ab831b18849289e3 P e7b2f19d10334c528ecde9b8263c4ab0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:33:55.937150 14700 raft_consensus.cc:2454] T a433e6521faa4f76ab831b18849289e3 P 784295b44bb04262a2d6152512ad3c43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8be4ea8f027d452290ed47e52d7a6ae0 in term 1.
I20260929 20:33:55.937553 14836 raft_consensus.cc:2454] T a433e6521faa4f76ab831b18849289e3 P e7b2f19d10334c528ecde9b8263c4ab0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8be4ea8f027d452290ed47e52d7a6ae0 in term 1.
I20260929 20:33:55.937681 14496 leader_election.cc:302] T a433e6521faa4f76ab831b18849289e3 P 8be4ea8f027d452290ed47e52d7a6ae0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 784295b44bb04262a2d6152512ad3c43, 8be4ea8f027d452290ed47e52d7a6ae0; no voters:
I20260929 20:33:55.938010 14910 raft_consensus.cc:2781] T a433e6521faa4f76ab831b18849289e3 P 8be4ea8f027d452290ed47e52d7a6ae0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:33:55.938498 14910 raft_consensus.cc:686] T a433e6521faa4f76ab831b18849289e3 P 8be4ea8f027d452290ed47e52d7a6ae0 [term 1 LEADER]: Becoming Leader. State: Replica: 8be4ea8f027d452290ed47e52d7a6ae0, State: Running, Role: LEADER
I20260929 20:33:55.939042 14910 consensus_queue.cc:237] T a433e6521faa4f76ab831b18849289e3 P 8be4ea8f027d452290ed47e52d7a6ae0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "784295b44bb04262a2d6152512ad3c43" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 39717 } } peers { permanent_uuid: "e7b2f19d10334c528ecde9b8263c4ab0" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 42291 } } peers { permanent_uuid: "8be4ea8f027d452290ed47e52d7a6ae0" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 39649 } }
I20260929 20:33:55.944290 14416 catalog_manager.cc:5495] T a433e6521faa4f76ab831b18849289e3 P 8be4ea8f027d452290ed47e52d7a6ae0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8be4ea8f027d452290ed47e52d7a6ae0 (127.28.244.129). New cstate: current_term: 1 leader_uuid: "8be4ea8f027d452290ed47e52d7a6ae0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "784295b44bb04262a2d6152512ad3c43" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 39717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7b2f19d10334c528ecde9b8263c4ab0" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 42291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8be4ea8f027d452290ed47e52d7a6ae0" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 39649 } health_report { overall_health: HEALTHY } } }
W20260929 20:33:55.946354 14883 tablet.cc:2369] T a433e6521faa4f76ab831b18849289e3 P e7b2f19d10334c528ecde9b8263c4ab0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:33:55.980625 14898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:55.974450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41616 (local address 127.28.244.129:39649)
0929 20:33:55.974709 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:55.974715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:55.974871 (+ 156us) server_negotiation.cc:407] Connection header received
0929 20:33:55.975136 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:55.975140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:55.975260 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:55.975432 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:33:55.975880 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:55.976653 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:55.978201 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:55.978649 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:55.978803 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:55.979387 (+ 584us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:33:55.979972 (+ 585us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:33:55.980150 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:55.980241 (+ 91us) server_negotiation.cc:299] Negotiation successful
0929 20:33:55.980336 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 20:33:55.989614 14836 raft_consensus.cc:1261] T a433e6521faa4f76ab831b18849289e3 P e7b2f19d10334c528ecde9b8263c4ab0 [term 1 FOLLOWER]: Refusing update from remote peer 8be4ea8f027d452290ed47e52d7a6ae0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:33:55.989636 14700 raft_consensus.cc:1261] T a433e6521faa4f76ab831b18849289e3 P 784295b44bb04262a2d6152512ad3c43 [term 1 FOLLOWER]: Refusing update from remote peer 8be4ea8f027d452290ed47e52d7a6ae0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:33:55.990707 14913 consensus_queue.cc:1035] T a433e6521faa4f76ab831b18849289e3 P 8be4ea8f027d452290ed47e52d7a6ae0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "784295b44bb04262a2d6152512ad3c43" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 39717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:33:55.991233 14910 consensus_queue.cc:1035] T a433e6521faa4f76ab831b18849289e3 P 8be4ea8f027d452290ed47e52d7a6ae0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7b2f19d10334c528ecde9b8263c4ab0" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 42291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:33:56.004680 14916 mvcc.cc:205] Tried to move back new op lower bound from 7334764691402563584 to 7334764691210465280. Current Snapshot: MvccSnapshot[applied={T|T < 7334764691402563584}]
I20260929 20:33:56.020967 14415 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:33:56.021281 14415 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:33:56.025722 14415 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 06d12dc5acff46548f0e3003d5ca9b5d: Sending DeleteTablet for 3 replicas of tablet a433e6521faa4f76ab831b18849289e3
I20260929 20:33:56.027155 14680 tablet_service.cc:1514] Processing DeleteTablet for tablet a433e6521faa4f76ab831b18849289e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:33:56 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:41948
I20260929 20:33:56.027197 14544 tablet_service.cc:1514] Processing DeleteTablet for tablet a433e6521faa4f76ab831b18849289e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:33:56 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:41600
I20260929 20:33:56.028162 14926 tablet_replica.cc:330] T a433e6521faa4f76ab831b18849289e3 P 784295b44bb04262a2d6152512ad3c43: stopping tablet replica
I20260929 20:33:56.028663 14926 raft_consensus.cc:2229] T a433e6521faa4f76ab831b18849289e3 P 784295b44bb04262a2d6152512ad3c43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:33:56.028864 14816 tablet_service.cc:1514] Processing DeleteTablet for tablet a433e6521faa4f76ab831b18849289e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:33:56 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:38042
I20260929 20:33:56.029217 14926 raft_consensus.cc:2258] T a433e6521faa4f76ab831b18849289e3 P 784295b44bb04262a2d6152512ad3c43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:33:56.030220 14928 tablet_replica.cc:330] T a433e6521faa4f76ab831b18849289e3 P e7b2f19d10334c528ecde9b8263c4ab0: stopping tablet replica
I20260929 20:33:56.030670 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 14480
I20260929 20:33:56.030715 14928 raft_consensus.cc:2229] T a433e6521faa4f76ab831b18849289e3 P e7b2f19d10334c528ecde9b8263c4ab0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:33:56.031244 14928 raft_consensus.cc:2258] T a433e6521faa4f76ab831b18849289e3 P e7b2f19d10334c528ecde9b8263c4ab0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:33:56.033612 14928 ts_tablet_manager.cc:1905] T a433e6521faa4f76ab831b18849289e3 P e7b2f19d10334c528ecde9b8263c4ab0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:33:56.037621 14926 ts_tablet_manager.cc:1905] T a433e6521faa4f76ab831b18849289e3 P 784295b44bb04262a2d6152512ad3c43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:33:56.040060 14928 ts_tablet_manager.cc:1918] T a433e6521faa4f76ab831b18849289e3 P e7b2f19d10334c528ecde9b8263c4ab0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:33:56.040256 14928 log.cc:1196] T a433e6521faa4f76ab831b18849289e3 P e7b2f19d10334c528ecde9b8263c4ab0: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-2/wal/wals/a433e6521faa4f76ab831b18849289e3
I20260929 20:33:56.040795 14928 ts_tablet_manager.cc:1939] T a433e6521faa4f76ab831b18849289e3 P e7b2f19d10334c528ecde9b8263c4ab0: Deleting consensus metadata
I20260929 20:33:56.042821 14400 catalog_manager.cc:4841] TS e7b2f19d10334c528ecde9b8263c4ab0 (127.28.244.131:42291): tablet a433e6521faa4f76ab831b18849289e3 (table test-table [id=4a5b2cf833914c85acf1bccb7863bba0]) successfully deleted
I20260929 20:33:56.044775 14927 tablet_replica.cc:330] T a433e6521faa4f76ab831b18849289e3 P 8be4ea8f027d452290ed47e52d7a6ae0: stopping tablet replica
I20260929 20:33:56.045269 14927 raft_consensus.cc:2229] T a433e6521faa4f76ab831b18849289e3 P 8be4ea8f027d452290ed47e52d7a6ae0 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:33:56.045897 14926 ts_tablet_manager.cc:1918] T a433e6521faa4f76ab831b18849289e3 P 784295b44bb04262a2d6152512ad3c43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:33:56.046043 14927 raft_consensus.cc:2258] T a433e6521faa4f76ab831b18849289e3 P 8be4ea8f027d452290ed47e52d7a6ae0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:33:56.046133 14926 log.cc:1196] T a433e6521faa4f76ab831b18849289e3 P 784295b44bb04262a2d6152512ad3c43: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-1/wal/wals/a433e6521faa4f76ab831b18849289e3
I20260929 20:33:56.047381 14926 ts_tablet_manager.cc:1939] T a433e6521faa4f76ab831b18849289e3 P 784295b44bb04262a2d6152512ad3c43: Deleting consensus metadata
I20260929 20:33:56.048362 14927 ts_tablet_manager.cc:1905] T a433e6521faa4f76ab831b18849289e3 P 8be4ea8f027d452290ed47e52d7a6ae0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:33:56.049444 14400 catalog_manager.cc:4841] TS 784295b44bb04262a2d6152512ad3c43 (127.28.244.130:39717): tablet a433e6521faa4f76ab831b18849289e3 (table test-table [id=4a5b2cf833914c85acf1bccb7863bba0]) successfully deleted
I20260929 20:33:56.054901 14927 ts_tablet_manager.cc:1918] T a433e6521faa4f76ab831b18849289e3 P 8be4ea8f027d452290ed47e52d7a6ae0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:33:56.055135 14927 log.cc:1196] T a433e6521faa4f76ab831b18849289e3 P 8be4ea8f027d452290ed47e52d7a6ae0: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713721201454-29650-0/minicluster-data/ts-0/wal/wals/a433e6521faa4f76ab831b18849289e3
I20260929 20:33:56.055655 14927 ts_tablet_manager.cc:1939] T a433e6521faa4f76ab831b18849289e3 P 8be4ea8f027d452290ed47e52d7a6ae0: Deleting consensus metadata
Sep 29 20:33:56 dist-test-slave-p29l krb5kdc[14371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.129@KRBTEST.COM
I20260929 20:33:56.057871 14403 catalog_manager.cc:4841] TS 8be4ea8f027d452290ed47e52d7a6ae0 (127.28.244.129:39649): tablet a433e6521faa4f76ab831b18849289e3 (table test-table [id=4a5b2cf833914c85acf1bccb7863bba0]) successfully deleted
I20260929 20:33:56.066057 14898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:56.033004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41628 (local address 127.28.244.129:39649)
0929 20:33:56.033475 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:56.033482 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:56.043674 (+ 10192us) server_negotiation.cc:407] Connection header received
0929 20:33:56.044123 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:56.044129 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:56.044343 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:56.044480 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:33:56.046422 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:56.047793 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:56.050015 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:56.050824 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:56.050963 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:56.057441 (+ 6478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:33:56.057463 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:33:56.057486 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:33:56.057550 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:33:56.062401 (+ 4851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:56.063299 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:56.063311 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:56.063318 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:56.063538 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:56.064180 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:56.064186 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:56.064191 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:56.064707 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:33:56.065072 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:56.065317 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 20:33:56.065608 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342}
I20260929 20:33:56.066956 14605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:33:56.171727 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 14480
I20260929 20:33:56.192193 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 14617
Sep 29 20:33:56 dist-test-slave-p29l krb5kdc[14371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.130@KRBTEST.COM
I20260929 20:33:56.209197 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:56.193143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41954 (local address 127.28.244.130:39717)
0929 20:33:56.193348 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:56.193356 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:56.193388 (+ 32us) server_negotiation.cc:407] Connection header received
0929 20:33:56.193583 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:56.193589 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:56.193679 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:56.193789 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:33:56.195054 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:56.195890 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:56.197308 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:56.197734 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:56.197875 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:56.202827 (+ 4952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:33:56.202845 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:33:56.202862 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:33:56.202904 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:33:56.205991 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:56.206861 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:56.206871 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:56.206874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:56.207008 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:56.207433 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:56.207440 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:56.207448 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:56.207934 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:33:56.208264 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:56.208601 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 20:33:56.208836 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 20:33:56.209945 14741 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:33:56.301609 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 14617
I20260929 20:33:56.320143 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 14752
Sep 29 20:33:56 dist-test-slave-p29l krb5kdc[14371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.131@KRBTEST.COM
I20260929 20:33:56.337334 14900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:56.320936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38048 (local address 127.28.244.131:42291)
0929 20:33:56.321184 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:56.321192 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:56.321216 (+ 24us) server_negotiation.cc:407] Connection header received
0929 20:33:56.321487 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:56.321496 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:56.321577 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:56.321683 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:33:56.323070 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:56.323885 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:56.325261 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:56.325682 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:56.325822 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:56.330821 (+ 4999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:33:56.330840 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:33:56.330857 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:33:56.330907 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:33:56.334092 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:56.334905 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:56.334913 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:56.334915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:56.335047 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:33:56.335456 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:33:56.335461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:33:56.335464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:33:56.336008 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:33:56.336338 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:56.336727 (+ 389us) server_negotiation.cc:299] Negotiation successful
0929 20:33:56.336972 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260929 20:33:56.338277 14876 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:33:56.434233 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 14752
I20260929 20:33:56.452384 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 14384
I20260929 20:33:56.453547 14448 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:33:56.566466 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 14384
2026-09-29T20:33:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4598 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T20:33:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:33:56Z Disabled control of system clock
I20260929 20:33:56.653844 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:41531
--webserver_interface=127.28.244.190
--webserver_port=0
--builtin_ntp_servers=127.28.244.148:43055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:41531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 20:33:57.008780 14937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:33:57.009115 14937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:33:57.009197 14937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:33:57.016889 14937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:33:57.017014 14937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:33:57.017053 14937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:33:57.017113 14937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:33:57.017158 14937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:33:57.026811 14937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:43055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:41531
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:41531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:33:57.028175 14937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:33:57.030113 14937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:33:57.042496 14943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:33:57.042517 14942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:33:57.042557 14945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:33:57.043520 14937 server_base.cc:1034] running on GCE node
I20260929 20:33:57.044497 14937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:33:57.046150 14937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:33:57.047412 14937 hybrid_clock.cc:648] HybridClock initialized: now 1790714037047361 us; error 58 us; skew 500 ppm
I20260929 20:33:57.047843 14937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:33:57.055447 14937 webserver.cc:460] Webserver started at http://127.28.244.190:32825/ using document root <none> and password file <none>
I20260929 20:33:57.056342 14937 fs_manager.cc:359] Metadata directory not provided
I20260929 20:33:57.056493 14937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:33:57.056838 14937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:33:57.060200 14937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "ac1f33774ec84b1e966e251d3a255e36"
format_stamp: "Formatted at 2026-09-29 20:33:57 on dist-test-slave-p29l"
server_key: "6a1ddb1f186688c9cb8cacdbde3b3cfd"
server_key_iv: "51948ba5e8ef286bcc7e58aebd4583a6"
server_key_version: "encryptionkey@0"
I20260929 20:33:57.061097 14937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "ac1f33774ec84b1e966e251d3a255e36"
format_stamp: "Formatted at 2026-09-29 20:33:57 on dist-test-slave-p29l"
server_key: "6a1ddb1f186688c9cb8cacdbde3b3cfd"
server_key_iv: "51948ba5e8ef286bcc7e58aebd4583a6"
server_key_version: "encryptionkey@0"
I20260929 20:33:57.066787 14937 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:33:57.070784 14951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:33:57.072332 14937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:33:57.072552 14937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "ac1f33774ec84b1e966e251d3a255e36"
format_stamp: "Formatted at 2026-09-29 20:33:57 on dist-test-slave-p29l"
server_key: "6a1ddb1f186688c9cb8cacdbde3b3cfd"
server_key_iv: "51948ba5e8ef286bcc7e58aebd4583a6"
server_key_version: "encryptionkey@0"
I20260929 20:33:57.072826 14937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:33:57.096101 14937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:33:57.097043 14937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:33:57.097360 14937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:33:57.117806 14937 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:41531
I20260929 20:33:57.117830 15002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:41531 every 8 connection(s)
I20260929 20:33:57.119326 14937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:33:57.120518 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 14937
I20260929 20:33:57.120729 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:33:57.121373 29650 external_mini_cluster.cc:1428] Setting key 4037f135324ca2e3e1a686f1f41116d7
I20260929 20:33:57.124840 15003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:33:57.135941 15006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:57.123651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38786 (local address 127.28.244.190:41531)
0929 20:33:57.124730 (+ 1079us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:57.124741 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:57.124786 (+ 45us) server_negotiation.cc:407] Connection header received
0929 20:33:57.126549 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:57.126572 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:57.127048 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:57.127502 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:33:57.128144 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:57.129508 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:57.130974 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:57.131700 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:57.131942 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:57.133273 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:33:57.133281 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:33:57.133297 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:33:57.133902 (+ 605us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:33:57.133992 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:33:57.134015 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:33:57.134161 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:57.134412 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 20:33:57.134799 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":374,"threads_started":1}
I20260929 20:33:57.137857 15003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac1f33774ec84b1e966e251d3a255e36: Bootstrap starting.
I20260929 20:33:57.141741 15003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac1f33774ec84b1e966e251d3a255e36: Neither blocks nor log segments found. Creating new log.
I20260929 20:33:57.143154 15003 log.cc:824] T 00000000000000000000000000000000 P ac1f33774ec84b1e966e251d3a255e36: Log is configured to *not* fsync() on all Append() calls
I20260929 20:33:57.146478 15003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac1f33774ec84b1e966e251d3a255e36: No bootstrap required, opened a new log
I20260929 20:33:57.154631 15003 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac1f33774ec84b1e966e251d3a255e36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac1f33774ec84b1e966e251d3a255e36" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 41531 } }
I20260929 20:33:57.155083 15003 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac1f33774ec84b1e966e251d3a255e36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:33:57.155194 15003 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac1f33774ec84b1e966e251d3a255e36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac1f33774ec84b1e966e251d3a255e36, State: Initialized, Role: FOLLOWER
I20260929 20:33:57.155843 15003 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac1f33774ec84b1e966e251d3a255e36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac1f33774ec84b1e966e251d3a255e36" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 41531 } }
I20260929 20:33:57.156126 15003 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac1f33774ec84b1e966e251d3a255e36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:33:57.156273 15003 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac1f33774ec84b1e966e251d3a255e36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:33:57.156478 15003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac1f33774ec84b1e966e251d3a255e36 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:33:57.159030 15003 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac1f33774ec84b1e966e251d3a255e36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac1f33774ec84b1e966e251d3a255e36" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 41531 } }
I20260929 20:33:57.159731 15003 leader_election.cc:302] T 00000000000000000000000000000000 P ac1f33774ec84b1e966e251d3a255e36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac1f33774ec84b1e966e251d3a255e36; no voters:
I20260929 20:33:57.160491 15003 leader_election.cc:288] T 00000000000000000000000000000000 P ac1f33774ec84b1e966e251d3a255e36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:33:57.160669 15008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac1f33774ec84b1e966e251d3a255e36 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:33:57.161392 15008 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac1f33774ec84b1e966e251d3a255e36 [term 1 LEADER]: Becoming Leader. State: Replica: ac1f33774ec84b1e966e251d3a255e36, State: Running, Role: LEADER
I20260929 20:33:57.161916 15008 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac1f33774ec84b1e966e251d3a255e36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac1f33774ec84b1e966e251d3a255e36" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 41531 } }
I20260929 20:33:57.162735 15003 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac1f33774ec84b1e966e251d3a255e36 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:33:57.165040 15010 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac1f33774ec84b1e966e251d3a255e36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac1f33774ec84b1e966e251d3a255e36. Latest consensus state: current_term: 1 leader_uuid: "ac1f33774ec84b1e966e251d3a255e36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac1f33774ec84b1e966e251d3a255e36" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 41531 } } }
I20260929 20:33:57.165611 15010 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac1f33774ec84b1e966e251d3a255e36 [sys.catalog]: This master's current role is: LEADER
I20260929 20:33:57.165452 15009 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac1f33774ec84b1e966e251d3a255e36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac1f33774ec84b1e966e251d3a255e36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac1f33774ec84b1e966e251d3a255e36" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 41531 } } }
I20260929 20:33:57.165756 15009 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac1f33774ec84b1e966e251d3a255e36 [sys.catalog]: This master's current role is: LEADER
I20260929 20:33:57.167171 15017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:33:57.173408 15017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:33:57.184047 15017 catalog_manager.cc:1347] Generated new cluster ID: 837562effd5c4bd7a0d1fea833db8655
I20260929 20:33:57.184204 15017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:33:57.203677 15017 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:33:57.205132 15017 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:33:57.215364 15017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac1f33774ec84b1e966e251d3a255e36: Generated new TSK 0
I20260929 20:33:57.216393 15017 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:33:57.226778 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.129:0
--local_ip_for_outbound_sockets=127.28.244.129
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:41531
--builtin_ntp_servers=127.28.244.148:43055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 20:33:57.579276 15027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:33:57.579589 15027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:33:57.579675 15027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:33:57.587221 15027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:33:57.587338 15027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:33:57.587445 15027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.129
I20260929 20:33:57.597851 15027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:43055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:41531
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.129
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:33:57.599574 15027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:33:57.601590 15027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:33:57.615491 15032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:33:57.615582 15035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:33:57.615585 15033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:33:57.616519 15027 server_base.cc:1034] running on GCE node
I20260929 20:33:57.617327 15027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:33:57.618527 15027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:33:57.619760 15027 hybrid_clock.cc:648] HybridClock initialized: now 1790714037619703 us; error 55 us; skew 500 ppm
I20260929 20:33:57.620210 15027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:33:57.622925 15027 webserver.cc:460] Webserver started at http://127.28.244.129:34275/ using document root <none> and password file <none>
I20260929 20:33:57.623674 15027 fs_manager.cc:359] Metadata directory not provided
I20260929 20:33:57.623808 15027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:33:57.624217 15027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:33:57.627250 15027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-0/data/instance:
uuid: "e544284c195144499822f5dded5c0cbd"
format_stamp: "Formatted at 2026-09-29 20:33:57 on dist-test-slave-p29l"
server_key: "4af0c9107da97d0db72317a7bf2f7f7f"
server_key_iv: "62e80f89cf055f2cd7a77f962c22ffa7"
server_key_version: "encryptionkey@0"
I20260929 20:33:57.628181 15027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance:
uuid: "e544284c195144499822f5dded5c0cbd"
format_stamp: "Formatted at 2026-09-29 20:33:57 on dist-test-slave-p29l"
server_key: "4af0c9107da97d0db72317a7bf2f7f7f"
server_key_iv: "62e80f89cf055f2cd7a77f962c22ffa7"
server_key_version: "encryptionkey@0"
I20260929 20:33:57.634150 15027 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260929 20:33:57.638175 15041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:33:57.639637 15027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:33:57.639943 15027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-0/wal
uuid: "e544284c195144499822f5dded5c0cbd"
format_stamp: "Formatted at 2026-09-29 20:33:57 on dist-test-slave-p29l"
server_key: "4af0c9107da97d0db72317a7bf2f7f7f"
server_key_iv: "62e80f89cf055f2cd7a77f962c22ffa7"
server_key_version: "encryptionkey@0"
I20260929 20:33:57.640193 15027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:33:57.669210 15027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:33:57.670142 15027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:33:57.670461 15027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:33:57.671509 15027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:33:57.673214 15027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:33:57.673344 15027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:33:57.673478 15027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:33:57.673564 15027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:33:57.708103 15027 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.129:42303
I20260929 20:33:57.708194 15155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.129:42303 every 8 connection(s)
I20260929 20:33:57.709509 15027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
I20260929 20:33:57.713521 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 15027
I20260929 20:33:57.713711 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance
I20260929 20:33:57.714267 29650 external_mini_cluster.cc:1428] Setting key 60dae33a578357279d093d8d95055555
I20260929 20:33:57.717180 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.130:0
--local_ip_for_outbound_sockets=127.28.244.130
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:41531
--builtin_ntp_servers=127.28.244.148:43055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 20:33:57.723570 15158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:57.712239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:39383 (local address 127.28.244.190:41531)
0929 20:33:57.712691 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:57.712696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:57.714854 (+ 2158us) server_negotiation.cc:407] Connection header received
0929 20:33:57.716560 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:57.716565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:57.716653 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:57.716794 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:33:57.718232 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:57.718954 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:57.720653 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:57.721066 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:57.721207 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:57.722666 (+ 1459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:33:57.722671 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:33:57.722674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:33:57.722695 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:33:57.722734 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:33:57.722746 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:33:57.722850 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:57.723086 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 20:33:57.723194 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":241,"threads_started":1}
I20260929 20:33:57.724673 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:57.712867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:41531 (local address 127.28.244.129:39383)
0929 20:33:57.713930 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:57.713980 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 20:33:57.716286 (+ 2306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:33:57.717050 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:33:57.717059 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:33:57.717157 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:33:57.718039 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:57.718070 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:57.719107 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:57.719113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:57.720494 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:57.720507 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:57.721316 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:57.721324 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:57.721566 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:57.722315 (+ 749us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:33:57.722318 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:33:57.722365 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:33:57.722451 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:33:57.722849 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:33:57.722864 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:33:57.722874 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 20:33:57.723077 (+ 203us) client_negotiation.cc:240] Negotiation successful
0929 20:33:57.723512 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":340,"threads_started":1}
I20260929 20:33:57.726310 15156 heartbeater.cc:347] Connected to a master server at 127.28.244.190:41531
I20260929 20:33:57.726657 15156 heartbeater.cc:464] Registering TS with master...
I20260929 20:33:57.727660 15156 heartbeater.cc:511] Master 127.28.244.190:41531 requested a full tablet report, sending...
I20260929 20:33:57.730270 14968 ts_manager.cc:194] Registered new tserver with Master: e544284c195144499822f5dded5c0cbd (127.28.244.129:42303)
I20260929 20:33:57.732635 14968 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.244.129:39383
W20260929 20:33:58.067611 15160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:33:58.068003 15160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:33:58.068097 15160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:33:58.075846 15160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:33:58.075965 15160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:33:58.076136 15160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.130
I20260929 20:33:58.085800 15160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:43055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:41531
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.130
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:33:58.087155 15160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:33:58.089052 15160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:33:58.103542 15165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:33:58.103607 15168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:33:58.103660 15166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:33:58.109733 15160 server_base.cc:1034] running on GCE node
I20260929 20:33:58.110415 15160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:33:58.111456 15160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:33:58.112726 15160 hybrid_clock.cc:648] HybridClock initialized: now 1790714038112677 us; error 58 us; skew 500 ppm
I20260929 20:33:58.113096 15160 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:33:58.115633 15160 webserver.cc:460] Webserver started at http://127.28.244.130:34163/ using document root <none> and password file <none>
I20260929 20:33:58.116496 15160 fs_manager.cc:359] Metadata directory not provided
I20260929 20:33:58.116641 15160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:33:58.117022 15160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:33:58.120306 15160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-1/data/instance:
uuid: "bcf590549a534792bd5491600419519b"
format_stamp: "Formatted at 2026-09-29 20:33:58 on dist-test-slave-p29l"
server_key: "042eda9c5b60559913ca4b270b880d9e"
server_key_iv: "1d3152fa43f3f6b171588fef68ff6425"
server_key_version: "encryptionkey@0"
I20260929 20:33:58.121160 15160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance:
uuid: "bcf590549a534792bd5491600419519b"
format_stamp: "Formatted at 2026-09-29 20:33:58 on dist-test-slave-p29l"
server_key: "042eda9c5b60559913ca4b270b880d9e"
server_key_iv: "1d3152fa43f3f6b171588fef68ff6425"
server_key_version: "encryptionkey@0"
I20260929 20:33:58.126844 15160 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:33:58.130736 15174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:33:58.132345 15160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 20:33:58.132490 15160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-1/wal
uuid: "bcf590549a534792bd5491600419519b"
format_stamp: "Formatted at 2026-09-29 20:33:58 on dist-test-slave-p29l"
server_key: "042eda9c5b60559913ca4b270b880d9e"
server_key_iv: "1d3152fa43f3f6b171588fef68ff6425"
server_key_version: "encryptionkey@0"
I20260929 20:33:58.132682 15160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:33:58.158978 15160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:33:58.159900 15160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:33:58.160204 15160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:33:58.161190 15160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:33:58.162835 15160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:33:58.162946 15160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:33:58.163061 15160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:33:58.163133 15160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:33:58.198414 15160 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.130:38289
I20260929 20:33:58.198545 15289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.130:38289 every 8 connection(s)
I20260929 20:33:58.199791 15160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
I20260929 20:33:58.207358 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 15160
I20260929 20:33:58.207559 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance
I20260929 20:33:58.208232 29650 external_mini_cluster.cc:1428] Setting key 2e04f0b6714a7fb339e0610d21a227b4
I20260929 20:33:58.211700 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.131:0
--local_ip_for_outbound_sockets=127.28.244.131
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:41531
--builtin_ntp_servers=127.28.244.148:43055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 20:33:58.214200 15158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:58.202571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:38479 (local address 127.28.244.190:41531)
0929 20:33:58.202759 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:58.202763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:58.204217 (+ 1454us) server_negotiation.cc:407] Connection header received
0929 20:33:58.206010 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:58.206014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:58.206090 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:58.206194 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:33:58.207322 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:58.208514 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:58.210796 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:58.211458 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:58.211597 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:58.213198 (+ 1601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:33:58.213208 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:33:58.213216 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:33:58.213245 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:33:58.213290 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:33:58.213309 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:33:58.213417 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:58.213652 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 20:33:58.213785 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:33:58.215153 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:58.203071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:41531 (local address 127.28.244.130:38479)
0929 20:33:58.204053 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:58.204097 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 20:33:58.205704 (+ 1607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:33:58.206403 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:33:58.206412 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:33:58.206514 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:33:58.207125 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:58.207144 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:58.208673 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:58.208677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:58.210642 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:58.210656 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:58.211690 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:58.211695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:58.212030 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:58.212813 (+ 783us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:33:58.212816 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:33:58.212861 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:33:58.212967 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:33:58.213419 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:33:58.213427 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:33:58.213433 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 20:33:58.213591 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 20:33:58.213926 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":279,"threads_started":1}
I20260929 20:33:58.216919 15290 heartbeater.cc:347] Connected to a master server at 127.28.244.190:41531
I20260929 20:33:58.217267 15290 heartbeater.cc:464] Registering TS with master...
I20260929 20:33:58.218163 15290 heartbeater.cc:511] Master 127.28.244.190:41531 requested a full tablet report, sending...
I20260929 20:33:58.219944 14968 ts_manager.cc:194] Registered new tserver with Master: bcf590549a534792bd5491600419519b (127.28.244.130:38289)
I20260929 20:33:58.220798 14968 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.244.130:38479
W20260929 20:33:58.560835 15293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:33:58.561151 15293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:33:58.561300 15293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:33:58.569178 15293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:33:58.569295 15293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:33:58.569464 15293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.131
I20260929 20:33:58.579344 15293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:43055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:41531
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.131
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:33:58.580768 15293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:33:58.582618 15293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:33:58.596356 15299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:33:58.596400 15301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:33:58.596426 15298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:33:58.597383 15293 server_base.cc:1034] running on GCE node
I20260929 20:33:58.598083 15293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:33:58.599208 15293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:33:58.600467 15293 hybrid_clock.cc:648] HybridClock initialized: now 1790714038600393 us; error 65 us; skew 500 ppm
I20260929 20:33:58.600916 15293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:33:58.604367 15293 webserver.cc:460] Webserver started at http://127.28.244.131:39801/ using document root <none> and password file <none>
I20260929 20:33:58.605176 15293 fs_manager.cc:359] Metadata directory not provided
I20260929 20:33:58.605324 15293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:33:58.605674 15293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:33:58.609028 15293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-2/data/instance:
uuid: "3aab96c10b3f4efd93c18550a7494fa5"
format_stamp: "Formatted at 2026-09-29 20:33:58 on dist-test-slave-p29l"
server_key: "7d4dd86db182ca9119593d2236b19b89"
server_key_iv: "96cd937fabe7d0e1825d6140b427a811"
server_key_version: "encryptionkey@0"
I20260929 20:33:58.609956 15293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance:
uuid: "3aab96c10b3f4efd93c18550a7494fa5"
format_stamp: "Formatted at 2026-09-29 20:33:58 on dist-test-slave-p29l"
server_key: "7d4dd86db182ca9119593d2236b19b89"
server_key_iv: "96cd937fabe7d0e1825d6140b427a811"
server_key_version: "encryptionkey@0"
I20260929 20:33:58.615959 15293 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260929 20:33:58.620322 15307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:33:58.622313 15293 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 20:33:58.622535 15293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-2/wal
uuid: "3aab96c10b3f4efd93c18550a7494fa5"
format_stamp: "Formatted at 2026-09-29 20:33:58 on dist-test-slave-p29l"
server_key: "7d4dd86db182ca9119593d2236b19b89"
server_key_iv: "96cd937fabe7d0e1825d6140b427a811"
server_key_version: "encryptionkey@0"
I20260929 20:33:58.622800 15293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:33:58.659510 15293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:33:58.660919 15293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:33:58.661317 15293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:33:58.662822 15293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:33:58.665284 15293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:33:58.665428 15293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:33:58.665570 15293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:33:58.665664 15293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:33:58.707962 15293 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.131:44865
I20260929 20:33:58.708117 15421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.131:44865 every 8 connection(s)
I20260929 20:33:58.709882 15293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
I20260929 20:33:58.713335 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 15293
I20260929 20:33:58.713523 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance
I20260929 20:33:58.714116 29650 external_mini_cluster.cc:1428] Setting key 5767f2479ba8e0bb337317081c9bb1a3
I20260929 20:33:58.733440 15158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:58.714052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:53733 (local address 127.28.244.190:41531)
0929 20:33:58.714313 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:58.714320 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:58.720919 (+ 6599us) server_negotiation.cc:407] Connection header received
0929 20:33:58.724160 (+ 3241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:58.724165 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:58.724242 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:58.724333 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:33:58.726013 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:58.726791 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:58.729763 (+ 2972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:58.730424 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:58.730526 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:58.732491 (+ 1965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:33:58.732497 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:33:58.732500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:33:58.732518 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:33:58.732550 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:33:58.732562 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:33:58.732636 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:58.733015 (+ 379us) server_negotiation.cc:299] Negotiation successful
0929 20:33:58.733114 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260929 20:33:58.735162 15424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:58.713415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:41531 (local address 127.28.244.131:53733)
0929 20:33:58.720648 (+ 7233us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:58.720707 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 20:33:58.723743 (+ 3036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:33:58.724648 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:33:58.724665 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:33:58.724834 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:33:58.725809 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:58.725839 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:58.726960 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:58.726969 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:58.729589 (+ 2620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:58.729611 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:58.730668 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:58.730678 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:58.731151 (+ 473us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:58.732050 (+ 899us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:33:58.732056 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:33:58.732124 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:33:58.732266 (+ 142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:33:58.732667 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:33:58.732680 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:33:58.732691 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 20:33:58.732928 (+ 237us) client_negotiation.cc:240] Negotiation successful
0929 20:33:58.733438 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6449,"thread_start_us":6251,"threads_started":1}
I20260929 20:33:58.737062 15422 heartbeater.cc:347] Connected to a master server at 127.28.244.190:41531
I20260929 20:33:58.737106 15156 heartbeater.cc:503] Master 127.28.244.190:41531 was elected leader, sending a full tablet report...
I20260929 20:33:58.737572 15422 heartbeater.cc:464] Registering TS with master...
I20260929 20:33:58.738847 15422 heartbeater.cc:511] Master 127.28.244.190:41531 requested a full tablet report, sending...
I20260929 20:33:58.740765 14968 ts_manager.cc:194] Registered new tserver with Master: 3aab96c10b3f4efd93c18550a7494fa5 (127.28.244.131:44865)
I20260929 20:33:58.742223 14968 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.244.131:53733
I20260929 20:33:58.743187 29650 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:33:58.756605 15158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:58.748600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39766 (local address 127.28.244.190:41531)
0929 20:33:58.748894 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:58.748901 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:58.749324 (+ 423us) server_negotiation.cc:407] Connection header received
0929 20:33:58.749542 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:58.749547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:58.749621 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:58.749718 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:33:58.750414 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:58.751199 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:58.753774 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:58.754228 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:58.754334 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:58.755740 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:33:58.755745 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:33:58.755748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:33:58.755769 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:33:58.755874 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:33:58.755895 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:33:58.755974 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:58.756161 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 20:33:58.756248 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260929 20:33:58.761940 14967 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:33:58.776811 14967 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:33:58.814828 15434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:58.792113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:42303 (local address 127.0.0.1:48356)
0929 20:33:58.792921 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:58.792965 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 20:33:58.793180 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:33:58.800143 (+ 6963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:33:58.800154 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:33:58.800194 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:33:58.800638 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:58.800656 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:58.804918 (+ 4262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:58.804926 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:58.809480 (+ 4554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:58.809502 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:58.812151 (+ 2649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:58.812161 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:58.813985 (+ 1824us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:58.814072 (+ 87us) client_negotiation.cc:769] Sending connection context
0929 20:33:58.814204 (+ 132us) client_negotiation.cc:240] Negotiation successful
0929 20:33:58.814437 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":23,"thread_start_us":831,"threads_started":1}
I20260929 20:33:58.815086 15436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:58.792761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48356 (local address 127.28.244.129:42303)
0929 20:33:58.797288 (+ 4527us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:58.797309 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:58.797350 (+ 41us) server_negotiation.cc:407] Connection header received
0929 20:33:58.797474 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:58.797483 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:58.797815 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:58.800932 (+ 3117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:33:58.801211 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:58.804718 (+ 3507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:58.809850 (+ 5132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:58.811884 (+ 2034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:58.813957 (+ 2073us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:58.814286 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:58.814372 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 20:33:58.814609 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4289,"thread_start_us":4135,"threads_started":1}
I20260929 20:33:58.820824 15437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:58.792738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:44865 (local address 127.0.0.1:39182)
0929 20:33:58.798446 (+ 5708us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:58.798469 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:33:58.798605 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:33:58.812696 (+ 14091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:33:58.812707 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:33:58.812754 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:33:58.813325 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:58.813344 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:58.815607 (+ 2263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:58.815612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:58.817855 (+ 2243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:58.817874 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:58.819897 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:58.819905 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:58.820111 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:58.820163 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 20:33:58.820239 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 20:33:58.820399 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5551,"mutex_wait_us":53,"thread_start_us":5403,"threads_started":1}
I20260929 20:33:58.820595 15091 tablet_service.cc:1467] Processing CreateTablet for tablet ac179b2c3b0e4310ae2e3620f7f92ddf (DEFAULT_TABLE table=test-table [id=7f6c6f341ff041eebad656c27ac540a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:33:58.822136 15435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:58.792296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39182 (local address 127.28.244.131:44865)
0929 20:33:58.811767 (+ 19471us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:58.811776 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:58.811836 (+ 60us) server_negotiation.cc:407] Connection header received
0929 20:33:58.811951 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:58.811959 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:58.812234 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:58.813530 (+ 1296us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:33:58.813834 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:58.815438 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:58.818184 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:58.819641 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:58.821261 (+ 1620us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:58.821492 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:58.821566 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 20:33:58.821747 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19264,"thread_start_us":20097,"threads_started":1}
I20260929 20:33:58.822240 15438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:58.796466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:38289 (local address 127.0.0.1:40574)
0929 20:33:58.807112 (+ 10646us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:58.807138 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 20:33:58.807317 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:33:58.808927 (+ 1610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:33:58.808936 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:33:58.808976 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:33:58.809610 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:58.809628 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:58.812393 (+ 2765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:58.812400 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:58.816695 (+ 4295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:58.816717 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:58.821188 (+ 4471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:58.821198 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:58.821416 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:58.821466 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 20:33:58.821562 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 20:33:58.821748 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10467,"mutex_wait_us":1,"thread_start_us":12160,"threads_started":1}
I20260929 20:33:58.823187 15091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac179b2c3b0e4310ae2e3620f7f92ddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:33:58.823207 15439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:58.796690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40574 (local address 127.28.244.130:38289)
0929 20:33:58.805594 (+ 8904us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:58.805607 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:58.807490 (+ 1883us) server_negotiation.cc:407] Connection header received
0929 20:33:58.807639 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:58.807652 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:58.808084 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:58.808340 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:33:58.809866 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:58.812215 (+ 2349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:58.817003 (+ 4788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:58.820552 (+ 3549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:58.820754 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:58.821037 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:58.822522 (+ 1485us) server_negotiation.cc:299] Negotiation successful
0929 20:33:58.822758 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8715,"thread_start_us":8604,"threads_started":1}
I20260929 20:33:58.826234 15356 tablet_service.cc:1467] Processing CreateTablet for tablet ac179b2c3b0e4310ae2e3620f7f92ddf (DEFAULT_TABLE table=test-table [id=7f6c6f341ff041eebad656c27ac540a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:33:58.827952 15356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac179b2c3b0e4310ae2e3620f7f92ddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:33:58.827754 15223 tablet_service.cc:1467] Processing CreateTablet for tablet ac179b2c3b0e4310ae2e3620f7f92ddf (DEFAULT_TABLE table=test-table [id=7f6c6f341ff041eebad656c27ac540a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:33:58.829428 15223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac179b2c3b0e4310ae2e3620f7f92ddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:33:58.849085 15440 tablet_bootstrap.cc:492] T ac179b2c3b0e4310ae2e3620f7f92ddf P e544284c195144499822f5dded5c0cbd: Bootstrap starting.
I20260929 20:33:58.851696 15441 tablet_bootstrap.cc:492] T ac179b2c3b0e4310ae2e3620f7f92ddf P 3aab96c10b3f4efd93c18550a7494fa5: Bootstrap starting.
I20260929 20:33:58.853706 15440 tablet_bootstrap.cc:654] T ac179b2c3b0e4310ae2e3620f7f92ddf P e544284c195144499822f5dded5c0cbd: Neither blocks nor log segments found. Creating new log.
I20260929 20:33:58.855224 15440 log.cc:824] T ac179b2c3b0e4310ae2e3620f7f92ddf P e544284c195144499822f5dded5c0cbd: Log is configured to *not* fsync() on all Append() calls
I20260929 20:33:58.857637 15441 tablet_bootstrap.cc:654] T ac179b2c3b0e4310ae2e3620f7f92ddf P 3aab96c10b3f4efd93c18550a7494fa5: Neither blocks nor log segments found. Creating new log.
I20260929 20:33:58.859246 15441 log.cc:824] T ac179b2c3b0e4310ae2e3620f7f92ddf P 3aab96c10b3f4efd93c18550a7494fa5: Log is configured to *not* fsync() on all Append() calls
I20260929 20:33:58.860901 15442 tablet_bootstrap.cc:492] T ac179b2c3b0e4310ae2e3620f7f92ddf P bcf590549a534792bd5491600419519b: Bootstrap starting.
I20260929 20:33:58.865422 15442 tablet_bootstrap.cc:654] T ac179b2c3b0e4310ae2e3620f7f92ddf P bcf590549a534792bd5491600419519b: Neither blocks nor log segments found. Creating new log.
I20260929 20:33:58.866034 15440 tablet_bootstrap.cc:492] T ac179b2c3b0e4310ae2e3620f7f92ddf P e544284c195144499822f5dded5c0cbd: No bootstrap required, opened a new log
I20260929 20:33:58.866509 15440 ts_tablet_manager.cc:1397] T ac179b2c3b0e4310ae2e3620f7f92ddf P e544284c195144499822f5dded5c0cbd: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.009s
I20260929 20:33:58.867008 15442 log.cc:824] T ac179b2c3b0e4310ae2e3620f7f92ddf P bcf590549a534792bd5491600419519b: Log is configured to *not* fsync() on all Append() calls
I20260929 20:33:58.868041 15441 tablet_bootstrap.cc:492] T ac179b2c3b0e4310ae2e3620f7f92ddf P 3aab96c10b3f4efd93c18550a7494fa5: No bootstrap required, opened a new log
I20260929 20:33:58.868381 15441 ts_tablet_manager.cc:1397] T ac179b2c3b0e4310ae2e3620f7f92ddf P 3aab96c10b3f4efd93c18550a7494fa5: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.004s
I20260929 20:33:58.877005 15442 tablet_bootstrap.cc:492] T ac179b2c3b0e4310ae2e3620f7f92ddf P bcf590549a534792bd5491600419519b: No bootstrap required, opened a new log
I20260929 20:33:58.877431 15442 ts_tablet_manager.cc:1397] T ac179b2c3b0e4310ae2e3620f7f92ddf P bcf590549a534792bd5491600419519b: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.008s
I20260929 20:33:58.879891 15440 raft_consensus.cc:348] T ac179b2c3b0e4310ae2e3620f7f92ddf P e544284c195144499822f5dded5c0cbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aab96c10b3f4efd93c18550a7494fa5" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44865 } } peers { permanent_uuid: "e544284c195144499822f5dded5c0cbd" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 42303 } } peers { permanent_uuid: "bcf590549a534792bd5491600419519b" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 38289 } }
I20260929 20:33:58.880554 15440 raft_consensus.cc:374] T ac179b2c3b0e4310ae2e3620f7f92ddf P e544284c195144499822f5dded5c0cbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:33:58.880707 15440 raft_consensus.cc:729] T ac179b2c3b0e4310ae2e3620f7f92ddf P e544284c195144499822f5dded5c0cbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e544284c195144499822f5dded5c0cbd, State: Initialized, Role: FOLLOWER
I20260929 20:33:58.881569 15440 consensus_queue.cc:260] T ac179b2c3b0e4310ae2e3620f7f92ddf P e544284c195144499822f5dded5c0cbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aab96c10b3f4efd93c18550a7494fa5" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44865 } } peers { permanent_uuid: "e544284c195144499822f5dded5c0cbd" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 42303 } } peers { permanent_uuid: "bcf590549a534792bd5491600419519b" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 38289 } }
I20260929 20:33:58.884433 15441 raft_consensus.cc:348] T ac179b2c3b0e4310ae2e3620f7f92ddf P 3aab96c10b3f4efd93c18550a7494fa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aab96c10b3f4efd93c18550a7494fa5" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44865 } } peers { permanent_uuid: "e544284c195144499822f5dded5c0cbd" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 42303 } } peers { permanent_uuid: "bcf590549a534792bd5491600419519b" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 38289 } }
I20260929 20:33:58.885020 15441 raft_consensus.cc:374] T ac179b2c3b0e4310ae2e3620f7f92ddf P 3aab96c10b3f4efd93c18550a7494fa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:33:58.885133 15441 raft_consensus.cc:729] T ac179b2c3b0e4310ae2e3620f7f92ddf P 3aab96c10b3f4efd93c18550a7494fa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aab96c10b3f4efd93c18550a7494fa5, State: Initialized, Role: FOLLOWER
I20260929 20:33:58.885921 15441 consensus_queue.cc:260] T ac179b2c3b0e4310ae2e3620f7f92ddf P 3aab96c10b3f4efd93c18550a7494fa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aab96c10b3f4efd93c18550a7494fa5" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44865 } } peers { permanent_uuid: "e544284c195144499822f5dded5c0cbd" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 42303 } } peers { permanent_uuid: "bcf590549a534792bd5491600419519b" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 38289 } }
I20260929 20:33:58.890231 15440 ts_tablet_manager.cc:1428] T ac179b2c3b0e4310ae2e3620f7f92ddf P e544284c195144499822f5dded5c0cbd: Time spent starting tablet: real 0.024s user 0.010s sys 0.010s
I20260929 20:33:58.890302 15442 raft_consensus.cc:348] T ac179b2c3b0e4310ae2e3620f7f92ddf P bcf590549a534792bd5491600419519b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aab96c10b3f4efd93c18550a7494fa5" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44865 } } peers { permanent_uuid: "e544284c195144499822f5dded5c0cbd" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 42303 } } peers { permanent_uuid: "bcf590549a534792bd5491600419519b" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 38289 } }
I20260929 20:33:58.890793 15442 raft_consensus.cc:374] T ac179b2c3b0e4310ae2e3620f7f92ddf P bcf590549a534792bd5491600419519b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:33:58.890892 15442 raft_consensus.cc:729] T ac179b2c3b0e4310ae2e3620f7f92ddf P bcf590549a534792bd5491600419519b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf590549a534792bd5491600419519b, State: Initialized, Role: FOLLOWER
I20260929 20:33:58.891654 15442 consensus_queue.cc:260] T ac179b2c3b0e4310ae2e3620f7f92ddf P bcf590549a534792bd5491600419519b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aab96c10b3f4efd93c18550a7494fa5" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44865 } } peers { permanent_uuid: "e544284c195144499822f5dded5c0cbd" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 42303 } } peers { permanent_uuid: "bcf590549a534792bd5491600419519b" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 38289 } }
I20260929 20:33:58.893911 15422 heartbeater.cc:503] Master 127.28.244.190:41531 was elected leader, sending a full tablet report...
I20260929 20:33:58.894302 15441 ts_tablet_manager.cc:1428] T ac179b2c3b0e4310ae2e3620f7f92ddf P 3aab96c10b3f4efd93c18550a7494fa5: Time spent starting tablet: real 0.026s user 0.018s sys 0.005s
I20260929 20:33:58.898589 15290 heartbeater.cc:503] Master 127.28.244.190:41531 was elected leader, sending a full tablet report...
I20260929 20:33:58.899338 15442 ts_tablet_manager.cc:1428] T ac179b2c3b0e4310ae2e3620f7f92ddf P bcf590549a534792bd5491600419519b: Time spent starting tablet: real 0.022s user 0.016s sys 0.004s
W20260929 20:33:58.952117 15291 tablet.cc:2369] T ac179b2c3b0e4310ae2e3620f7f92ddf P bcf590549a534792bd5491600419519b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:33:58.962023 15157 tablet.cc:2369] T ac179b2c3b0e4310ae2e3620f7f92ddf P e544284c195144499822f5dded5c0cbd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:33:58.970299 15423 tablet.cc:2369] T ac179b2c3b0e4310ae2e3620f7f92ddf P 3aab96c10b3f4efd93c18550a7494fa5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:33:59.037161 15448 raft_consensus.cc:482] T ac179b2c3b0e4310ae2e3620f7f92ddf P bcf590549a534792bd5491600419519b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:33:59.037569 15448 raft_consensus.cc:504] T ac179b2c3b0e4310ae2e3620f7f92ddf P bcf590549a534792bd5491600419519b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aab96c10b3f4efd93c18550a7494fa5" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44865 } } peers { permanent_uuid: "e544284c195144499822f5dded5c0cbd" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 42303 } } peers { permanent_uuid: "bcf590549a534792bd5491600419519b" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 38289 } }
I20260929 20:33:59.039404 15448 leader_election.cc:288] T ac179b2c3b0e4310ae2e3620f7f92ddf P bcf590549a534792bd5491600419519b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aab96c10b3f4efd93c18550a7494fa5 (127.28.244.131:44865), e544284c195144499822f5dded5c0cbd (127.28.244.129:42303)
I20260929 20:33:59.048017 15436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:59.040015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:42085 (local address 127.28.244.129:42303)
0929 20:33:59.040210 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:59.040219 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:59.040854 (+ 635us) server_negotiation.cc:407] Connection header received
0929 20:33:59.041209 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:59.041216 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:59.041316 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:59.041434 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:33:59.042157 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:59.042932 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:59.045270 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:59.046596 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:59.046779 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:59.046860 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:59.047604 (+ 744us) server_negotiation.cc:299] Negotiation successful
0929 20:33:59.047744 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 20:33:59.048038 15435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:59.039864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:48685 (local address 127.28.244.131:44865)
0929 20:33:59.040079 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:59.040086 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:59.040786 (+ 700us) server_negotiation.cc:407] Connection header received
0929 20:33:59.041035 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:59.041041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:59.041125 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:59.041223 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:33:59.042366 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:59.043118 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:59.045410 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:59.046629 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:59.046804 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:59.046867 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:59.047604 (+ 737us) server_negotiation.cc:299] Negotiation successful
0929 20:33:59.047741 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 20:33:59.048159 15450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:59.039904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:42303 (local address 127.28.244.130:42085)
0929 20:33:59.040714 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:59.040753 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 20:33:59.040968 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:33:59.041482 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:33:59.041486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:33:59.041510 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:33:59.041957 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:59.041979 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:59.043069 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:59.043076 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:59.045053 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:59.045065 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:59.046903 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:59.046912 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:59.047300 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:59.047434 (+ 134us) client_negotiation.cc:769] Sending connection context
0929 20:33:59.047534 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 20:33:59.047711 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":201,"thread_start_us":474,"threads_started":1}
I20260929 20:33:59.048460 15449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:59.039741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:44865 (local address 127.28.244.130:48685)
0929 20:33:59.040638 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:59.040658 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:33:59.040829 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:33:59.041258 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:33:59.041262 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:33:59.041295 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:33:59.042207 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:59.042219 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:59.043256 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:59.043262 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:59.045142 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:33:59.045160 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:33:59.046908 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:33:59.046914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:33:59.047303 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:59.047436 (+ 133us) client_negotiation.cc:769] Sending connection context
0929 20:33:59.047534 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 20:33:59.047711 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"mutex_wait_us":211,"thread_start_us":471,"threads_started":1}
I20260929 20:33:59.049310 15111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac179b2c3b0e4310ae2e3620f7f92ddf" candidate_uuid: "bcf590549a534792bd5491600419519b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e544284c195144499822f5dded5c0cbd" is_pre_election: true
I20260929 20:33:59.049450 15376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac179b2c3b0e4310ae2e3620f7f92ddf" candidate_uuid: "bcf590549a534792bd5491600419519b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aab96c10b3f4efd93c18550a7494fa5" is_pre_election: true
I20260929 20:33:59.049994 15111 raft_consensus.cc:2454] T ac179b2c3b0e4310ae2e3620f7f92ddf P e544284c195144499822f5dded5c0cbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf590549a534792bd5491600419519b in term 0.
I20260929 20:33:59.049994 15376 raft_consensus.cc:2454] T ac179b2c3b0e4310ae2e3620f7f92ddf P 3aab96c10b3f4efd93c18550a7494fa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf590549a534792bd5491600419519b in term 0.
I20260929 20:33:59.050810 15177 leader_election.cc:302] T ac179b2c3b0e4310ae2e3620f7f92ddf P bcf590549a534792bd5491600419519b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3aab96c10b3f4efd93c18550a7494fa5, bcf590549a534792bd5491600419519b, e544284c195144499822f5dded5c0cbd; no voters:
I20260929 20:33:59.051312 15448 raft_consensus.cc:2781] T ac179b2c3b0e4310ae2e3620f7f92ddf P bcf590549a534792bd5491600419519b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:33:59.051482 15448 raft_consensus.cc:482] T ac179b2c3b0e4310ae2e3620f7f92ddf P bcf590549a534792bd5491600419519b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:33:59.051607 15448 raft_consensus.cc:3037] T ac179b2c3b0e4310ae2e3620f7f92ddf P bcf590549a534792bd5491600419519b [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:33:59.054142 15448 raft_consensus.cc:504] T ac179b2c3b0e4310ae2e3620f7f92ddf P bcf590549a534792bd5491600419519b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aab96c10b3f4efd93c18550a7494fa5" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44865 } } peers { permanent_uuid: "e544284c195144499822f5dded5c0cbd" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 42303 } } peers { permanent_uuid: "bcf590549a534792bd5491600419519b" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 38289 } }
I20260929 20:33:59.054881 15448 leader_election.cc:288] T ac179b2c3b0e4310ae2e3620f7f92ddf P bcf590549a534792bd5491600419519b [CANDIDATE]: Term 1 election: Requested vote from peers 3aab96c10b3f4efd93c18550a7494fa5 (127.28.244.131:44865), e544284c195144499822f5dded5c0cbd (127.28.244.129:42303)
I20260929 20:33:59.055544 15376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac179b2c3b0e4310ae2e3620f7f92ddf" candidate_uuid: "bcf590549a534792bd5491600419519b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aab96c10b3f4efd93c18550a7494fa5"
I20260929 20:33:59.055544 15111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac179b2c3b0e4310ae2e3620f7f92ddf" candidate_uuid: "bcf590549a534792bd5491600419519b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e544284c195144499822f5dded5c0cbd"
I20260929 20:33:59.055788 15111 raft_consensus.cc:3037] T ac179b2c3b0e4310ae2e3620f7f92ddf P e544284c195144499822f5dded5c0cbd [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:33:59.055922 15376 raft_consensus.cc:3037] T ac179b2c3b0e4310ae2e3620f7f92ddf P 3aab96c10b3f4efd93c18550a7494fa5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:33:59.059193 15376 raft_consensus.cc:2454] T ac179b2c3b0e4310ae2e3620f7f92ddf P 3aab96c10b3f4efd93c18550a7494fa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf590549a534792bd5491600419519b in term 1.
I20260929 20:33:59.059193 15111 raft_consensus.cc:2454] T ac179b2c3b0e4310ae2e3620f7f92ddf P e544284c195144499822f5dded5c0cbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf590549a534792bd5491600419519b in term 1.
I20260929 20:33:59.059842 15178 leader_election.cc:302] T ac179b2c3b0e4310ae2e3620f7f92ddf P bcf590549a534792bd5491600419519b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3aab96c10b3f4efd93c18550a7494fa5, bcf590549a534792bd5491600419519b, e544284c195144499822f5dded5c0cbd; no voters:
I20260929 20:33:59.060278 15448 raft_consensus.cc:2781] T ac179b2c3b0e4310ae2e3620f7f92ddf P bcf590549a534792bd5491600419519b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:33:59.060822 15448 raft_consensus.cc:686] T ac179b2c3b0e4310ae2e3620f7f92ddf P bcf590549a534792bd5491600419519b [term 1 LEADER]: Becoming Leader. State: Replica: bcf590549a534792bd5491600419519b, State: Running, Role: LEADER
I20260929 20:33:59.061349 15448 consensus_queue.cc:237] T ac179b2c3b0e4310ae2e3620f7f92ddf P bcf590549a534792bd5491600419519b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aab96c10b3f4efd93c18550a7494fa5" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44865 } } peers { permanent_uuid: "e544284c195144499822f5dded5c0cbd" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 42303 } } peers { permanent_uuid: "bcf590549a534792bd5491600419519b" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 38289 } }
I20260929 20:33:59.066648 14967 catalog_manager.cc:5495] T ac179b2c3b0e4310ae2e3620f7f92ddf P bcf590549a534792bd5491600419519b reported cstate change: term changed from 0 to 1, leader changed from <none> to bcf590549a534792bd5491600419519b (127.28.244.130). New cstate: current_term: 1 leader_uuid: "bcf590549a534792bd5491600419519b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aab96c10b3f4efd93c18550a7494fa5" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e544284c195144499822f5dded5c0cbd" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 42303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf590549a534792bd5491600419519b" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 38289 } health_report { overall_health: HEALTHY } } }
I20260929 20:33:59.111917 15439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:59.106239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40590 (local address 127.28.244.130:38289)
0929 20:33:59.106434 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:59.106439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:59.106472 (+ 33us) server_negotiation.cc:407] Connection header received
0929 20:33:59.106680 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:59.106684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:59.106774 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:59.106919 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:33:59.107237 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:59.107992 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:59.109640 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:59.110100 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:59.110210 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:59.110752 (+ 542us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:33:59.111339 (+ 587us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:33:59.111455 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:59.111534 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 20:33:59.111606 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:33:59.121332 15376 raft_consensus.cc:1261] T ac179b2c3b0e4310ae2e3620f7f92ddf P 3aab96c10b3f4efd93c18550a7494fa5 [term 1 FOLLOWER]: Refusing update from remote peer bcf590549a534792bd5491600419519b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:33:59.121331 15111 raft_consensus.cc:1261] T ac179b2c3b0e4310ae2e3620f7f92ddf P e544284c195144499822f5dded5c0cbd [term 1 FOLLOWER]: Refusing update from remote peer bcf590549a534792bd5491600419519b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:33:59.122462 15451 consensus_queue.cc:1035] T ac179b2c3b0e4310ae2e3620f7f92ddf P bcf590549a534792bd5491600419519b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aab96c10b3f4efd93c18550a7494fa5" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 44865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:33:59.122826 15448 consensus_queue.cc:1035] T ac179b2c3b0e4310ae2e3620f7f92ddf P bcf590549a534792bd5491600419519b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e544284c195144499822f5dded5c0cbd" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 42303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:33:59.136439 15454 mvcc.cc:205] Tried to move back new op lower bound from 7334764704228970496 to 7334764703999234048. Current Snapshot: MvccSnapshot[applied={T|T < 7334764704228970496}]
I20260929 20:33:59.146943 15456 mvcc.cc:205] Tried to move back new op lower bound from 7334764704228970496 to 7334764703999234048. Current Snapshot: MvccSnapshot[applied={T|T < 7334764704228970496}]
I20260929 20:33:59.157908 14967 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:33:59.158190 14967 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:33:59.162119 14967 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac1f33774ec84b1e966e251d3a255e36: Sending DeleteTablet for 3 replicas of tablet ac179b2c3b0e4310ae2e3620f7f92ddf
I20260929 20:33:59.163561 15356 tablet_service.cc:1514] Processing DeleteTablet for tablet ac179b2c3b0e4310ae2e3620f7f92ddf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:33:59 UTC) from {username='slave'} at 127.0.0.1:39182
I20260929 20:33:59.163563 15091 tablet_service.cc:1514] Processing DeleteTablet for tablet ac179b2c3b0e4310ae2e3620f7f92ddf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:33:59 UTC) from {username='slave'} at 127.0.0.1:48356
I20260929 20:33:59.164266 15223 tablet_service.cc:1514] Processing DeleteTablet for tablet ac179b2c3b0e4310ae2e3620f7f92ddf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:33:59 UTC) from {username='slave'} at 127.0.0.1:40574
I20260929 20:33:59.166194 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 15027
I20260929 20:33:59.177692 15466 tablet_replica.cc:330] T ac179b2c3b0e4310ae2e3620f7f92ddf P bcf590549a534792bd5491600419519b: stopping tablet replica
I20260929 20:33:59.178588 15466 raft_consensus.cc:2229] T ac179b2c3b0e4310ae2e3620f7f92ddf P bcf590549a534792bd5491600419519b [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:33:59.179667 15466 raft_consensus.cc:2258] T ac179b2c3b0e4310ae2e3620f7f92ddf P bcf590549a534792bd5491600419519b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:33:59.183868 15466 ts_tablet_manager.cc:1905] T ac179b2c3b0e4310ae2e3620f7f92ddf P bcf590549a534792bd5491600419519b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:33:59.190452 15464 tablet_replica.cc:330] T ac179b2c3b0e4310ae2e3620f7f92ddf P e544284c195144499822f5dded5c0cbd: stopping tablet replica
I20260929 20:33:59.190997 15464 raft_consensus.cc:2229] T ac179b2c3b0e4310ae2e3620f7f92ddf P e544284c195144499822f5dded5c0cbd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:33:59.191591 15464 raft_consensus.cc:2258] T ac179b2c3b0e4310ae2e3620f7f92ddf P e544284c195144499822f5dded5c0cbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:33:59.192482 15466 ts_tablet_manager.cc:1918] T ac179b2c3b0e4310ae2e3620f7f92ddf P bcf590549a534792bd5491600419519b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:33:59.192765 15466 log.cc:1196] T ac179b2c3b0e4310ae2e3620f7f92ddf P bcf590549a534792bd5491600419519b: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-1/wal/wals/ac179b2c3b0e4310ae2e3620f7f92ddf
I20260929 20:33:59.193465 15466 ts_tablet_manager.cc:1939] T ac179b2c3b0e4310ae2e3620f7f92ddf P bcf590549a534792bd5491600419519b: Deleting consensus metadata
I20260929 20:33:59.194787 15465 tablet_replica.cc:330] T ac179b2c3b0e4310ae2e3620f7f92ddf P 3aab96c10b3f4efd93c18550a7494fa5: stopping tablet replica
I20260929 20:33:59.196274 14955 catalog_manager.cc:4841] TS bcf590549a534792bd5491600419519b (127.28.244.130:38289): tablet ac179b2c3b0e4310ae2e3620f7f92ddf (table test-table [id=7f6c6f341ff041eebad656c27ac540a7]) successfully deleted
I20260929 20:33:59.196960 15464 ts_tablet_manager.cc:1905] T ac179b2c3b0e4310ae2e3620f7f92ddf P e544284c195144499822f5dded5c0cbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:33:59.196992 15436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:59.167552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48360 (local address 127.28.244.129:42303)
0929 20:33:59.167744 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:59.167750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:59.185355 (+ 17605us) server_negotiation.cc:407] Connection header received
0929 20:33:59.185903 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:59.185910 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:59.186047 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:59.186127 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:33:59.186811 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:59.188013 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:59.192059 (+ 4046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:59.192776 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:59.192894 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:59.195174 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:33:59.195185 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:33:59.195203 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:33:59.196080 (+ 877us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:33:59.196147 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:33:59.196177 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:33:59.196299 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:59.196537 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 20:33:59.196672 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:33:59.198057 15151 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:33:59.198130 15465 raft_consensus.cc:2229] T ac179b2c3b0e4310ae2e3620f7f92ddf P 3aab96c10b3f4efd93c18550a7494fa5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:33:59.198746 15465 raft_consensus.cc:2258] T ac179b2c3b0e4310ae2e3620f7f92ddf P 3aab96c10b3f4efd93c18550a7494fa5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:33:59.201704 15465 ts_tablet_manager.cc:1905] T ac179b2c3b0e4310ae2e3620f7f92ddf P 3aab96c10b3f4efd93c18550a7494fa5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:33:59.211099 15465 ts_tablet_manager.cc:1918] T ac179b2c3b0e4310ae2e3620f7f92ddf P 3aab96c10b3f4efd93c18550a7494fa5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:33:59.211315 15465 log.cc:1196] T ac179b2c3b0e4310ae2e3620f7f92ddf P 3aab96c10b3f4efd93c18550a7494fa5: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713721201454-29650-0/minicluster-data/ts-2/wal/wals/ac179b2c3b0e4310ae2e3620f7f92ddf
I20260929 20:33:59.211855 15465 ts_tablet_manager.cc:1939] T ac179b2c3b0e4310ae2e3620f7f92ddf P 3aab96c10b3f4efd93c18550a7494fa5: Deleting consensus metadata
I20260929 20:33:59.214136 14954 catalog_manager.cc:4841] TS 3aab96c10b3f4efd93c18550a7494fa5 (127.28.244.131:44865): tablet ac179b2c3b0e4310ae2e3620f7f92ddf (table test-table [id=7f6c6f341ff041eebad656c27ac540a7]) successfully deleted
I20260929 20:33:59.301124 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 15027
W20260929 20:33:59.320871 14955 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.244.129:42303 (error 108)
I20260929 20:33:59.321167 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 15160
I20260929 20:33:59.323508 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:59.322726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:42303 (local address 127.0.0.1:48376)
0929 20:33:59.323208 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:59.323341 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:42303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":230,"threads_started":1}
W20260929 20:33:59.323928 14955 catalog_manager.cc:4584] TS e544284c195144499822f5dded5c0cbd (127.28.244.129:42303): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ac179b2c3b0e4310ae2e3620f7f92ddf: Network error: Client connection negotiation failed: client connection to 127.28.244.129:42303: connect: Connection refused (error 111)
I20260929 20:33:59.329996 15439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:59.322132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40604 (local address 127.28.244.130:38289)
0929 20:33:59.322340 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:59.322346 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:59.322369 (+ 23us) server_negotiation.cc:407] Connection header received
0929 20:33:59.322498 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:59.322504 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:59.322597 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:59.322714 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:33:59.323226 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:59.324667 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:59.326039 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:59.326794 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:59.326954 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:59.328366 (+ 1412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:33:59.328376 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:33:59.328400 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:33:59.329068 (+ 668us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:33:59.329128 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:33:59.329149 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:33:59.329246 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:59.329457 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 20:33:59.329556 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:33:59.330850 15285 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:33:59.377697 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:59.377145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:42303 (local address 127.0.0.1:48388)
0929 20:33:59.377393 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:59.377512 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:42303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 20:33:59.378964 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:59.378508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:42303 (local address 127.0.0.1:48392)
0929 20:33:59.378688 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:59.378777 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:42303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 20:33:59.439626 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:59.439040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:42303 (local address 127.0.0.1:48394)
0929 20:33:59.439307 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:59.439434 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:42303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260929 20:33:59.440994 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:59.440533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:42303 (local address 127.0.0.1:48410)
0929 20:33:59.440719 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:59.440819 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:42303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 20:33:59.445609 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 15160
I20260929 20:33:59.465776 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 15293
I20260929 20:33:59.473026 15435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:59.466612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39198 (local address 127.28.244.131:44865)
0929 20:33:59.466842 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 20:33:59.466846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:33:59.466929 (+ 83us) server_negotiation.cc:407] Connection header received
0929 20:33:59.467084 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:33:59.467088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:33:59.467169 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:33:59.467257 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:33:59.467611 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:59.468479 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:59.469901 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:33:59.470359 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:33:59.470485 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:33:59.471562 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:33:59.471571 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:33:59.471588 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:33:59.472315 (+ 727us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:33:59.472365 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:33:59.472388 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:33:59.472479 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 20:33:59.472606 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 20:33:59.472701 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 20:33:59.473719 15417 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:33:59.522413 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:59.521801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:42303 (local address 127.0.0.1:48418)
0929 20:33:59.522064 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:59.522182 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:42303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 20:33:59.523741 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:59.523223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:42303 (local address 127.0.0.1:48426)
0929 20:33:59.523451 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:59.523543 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:42303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260929 20:33:59.563933 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 15293
I20260929 20:33:59.581972 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 14937
I20260929 20:33:59.583060 14998 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:33:59.700527 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:33:59.699927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:42303 (local address 127.0.0.1:48430)
0929 20:33:59.700163 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 20:33:59.700314 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:42303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 20:33:59.701144 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 14937
2026-09-29T20:33:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3128 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:33:59 dist-test-slave-p29l krb5kdc[15476](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:33:59 dist-test-slave-p29l krb5kdc[15476](info): set up 2 sockets
Sep 29 20:33:59 dist-test-slave-p29l krb5kdc[15476](info): commencing operation
krb5kdc: starting...
W20260929 20:34:00.831547 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.005s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:34:00 dist-test-slave-p29l krb5kdc[15476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:34:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:34:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.244.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
I20260929 20:34:01.099468 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:38051
--webserver_interface=127.28.244.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:46821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:38051
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:34:01.459154 15489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:34:01.459545 15489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:34:01.459722 15489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:34:01.467408 15489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:34:01.467589 15489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:34:01.467710 15489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:34:01.467856 15489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:34:01.467943 15489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:34:01.478271 15489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:46821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:38051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:38051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:34:01.479796 15489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:34:01.481235 15489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:34:01.494688 15494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:01.494663 15497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:01.494602 15495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:34:01.496115 15489 server_base.cc:1034] running on GCE node
I20260929 20:34:01.497004 15489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:34:01.498598 15489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:34:01.499922 15489 hybrid_clock.cc:648] HybridClock initialized: now 1790714041499901 us; error 63 us; skew 500 ppm
I20260929 20:34:01.500308 15489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:34:01 dist-test-slave-p29l krb5kdc[15476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714041, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:34:01.503655 15489 init.cc:377] Logged in from keytab as kudu/127.28.244.190@KRBTEST.COM (short username kudu)
I20260929 20:34:01.505379 15489 webserver.cc:460] Webserver started at http://127.28.244.190:46449/ using document root <none> and password file <none>
I20260929 20:34:01.506110 15489 fs_manager.cc:359] Metadata directory not provided
I20260929 20:34:01.506217 15489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:34:01.506481 15489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:34:01.509776 15489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "277a7093c46843c4bb8fe76c65323260"
format_stamp: "Formatted at 2026-09-29 20:34:01 on dist-test-slave-p29l"
server_key: "b35386919966848a45e034e3a5eda76e"
server_key_iv: "6a13b2669947c9d32667ef845ff23333"
server_key_version: "encryptionkey@0"
I20260929 20:34:01.510654 15489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "277a7093c46843c4bb8fe76c65323260"
format_stamp: "Formatted at 2026-09-29 20:34:01 on dist-test-slave-p29l"
server_key: "b35386919966848a45e034e3a5eda76e"
server_key_iv: "6a13b2669947c9d32667ef845ff23333"
server_key_version: "encryptionkey@0"
I20260929 20:34:01.516623 15489 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260929 20:34:01.520685 15504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:01.522248 15489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:34:01.522486 15489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "277a7093c46843c4bb8fe76c65323260"
format_stamp: "Formatted at 2026-09-29 20:34:01 on dist-test-slave-p29l"
server_key: "b35386919966848a45e034e3a5eda76e"
server_key_iv: "6a13b2669947c9d32667ef845ff23333"
server_key_version: "encryptionkey@0"
I20260929 20:34:01.522713 15489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:34:01.555262 15489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:34:01.560120 15489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:34:01.560461 15489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:34:01.580711 15489 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:38051
I20260929 20:34:01.580734 15556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:38051 every 8 connection(s)
I20260929 20:34:01.582186 15489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:34:01.586247 15557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:34:01.588084 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 15489
I20260929 20:34:01.588292 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:34:01.588737 29650 external_mini_cluster.cc:1428] Setting key 9979acbbb34caea06fca1ec98fc78d44
I20260929 20:34:01.601444 15557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 277a7093c46843c4bb8fe76c65323260: Bootstrap starting.
Sep 29 20:34:01 dist-test-slave-p29l krb5kdc[15476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:34:01.605933 15557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 277a7093c46843c4bb8fe76c65323260: Neither blocks nor log segments found. Creating new log.
I20260929 20:34:01.607355 15557 log.cc:824] T 00000000000000000000000000000000 P 277a7093c46843c4bb8fe76c65323260: Log is configured to *not* fsync() on all Append() calls
I20260929 20:34:01.611547 15557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 277a7093c46843c4bb8fe76c65323260: No bootstrap required, opened a new log
I20260929 20:34:01.616590 15560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:01.590781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52564 (local address 127.28.244.190:38051)
0929 20:34:01.591652 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:01.591673 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:01.591744 (+ 71us) server_negotiation.cc:407] Connection header received
0929 20:34:01.593559 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:01.593591 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:01.594157 (+ 566us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:01.594628 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:01.596035 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:01.598071 (+ 2036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:01.599568 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:01.600710 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:01.600913 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:01.606562 (+ 5649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:01.606574 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:01.606594 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:01.606648 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:01.611155 (+ 4507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:01.612084 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:01.612097 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:01.612107 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:01.612374 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:01.612810 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:01.612815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:01.612819 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:01.613639 (+ 820us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:01.614004 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:01.614374 (+ 370us) server_negotiation.cc:299] Negotiation successful
0929 20:34:01.614949 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":358,"threads_started":1}
I20260929 20:34:01.623307 15557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 277a7093c46843c4bb8fe76c65323260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "277a7093c46843c4bb8fe76c65323260" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 38051 } }
I20260929 20:34:01.623960 15557 raft_consensus.cc:374] T 00000000000000000000000000000000 P 277a7093c46843c4bb8fe76c65323260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:34:01.624086 15557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 277a7093c46843c4bb8fe76c65323260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 277a7093c46843c4bb8fe76c65323260, State: Initialized, Role: FOLLOWER
I20260929 20:34:01.624872 15557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 277a7093c46843c4bb8fe76c65323260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "277a7093c46843c4bb8fe76c65323260" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 38051 } }
I20260929 20:34:01.625209 15557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 277a7093c46843c4bb8fe76c65323260 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:34:01.625370 15557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 277a7093c46843c4bb8fe76c65323260 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:34:01.625579 15557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 277a7093c46843c4bb8fe76c65323260 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:34:01.628320 15557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 277a7093c46843c4bb8fe76c65323260 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "277a7093c46843c4bb8fe76c65323260" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 38051 } }
I20260929 20:34:01.628916 15557 leader_election.cc:302] T 00000000000000000000000000000000 P 277a7093c46843c4bb8fe76c65323260 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 277a7093c46843c4bb8fe76c65323260; no voters:
I20260929 20:34:01.629596 15557 leader_election.cc:288] T 00000000000000000000000000000000 P 277a7093c46843c4bb8fe76c65323260 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:34:01.629773 15562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 277a7093c46843c4bb8fe76c65323260 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:34:01.630688 15562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 277a7093c46843c4bb8fe76c65323260 [term 1 LEADER]: Becoming Leader. State: Replica: 277a7093c46843c4bb8fe76c65323260, State: Running, Role: LEADER
I20260929 20:34:01.631378 15562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 277a7093c46843c4bb8fe76c65323260 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "277a7093c46843c4bb8fe76c65323260" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 38051 } }
I20260929 20:34:01.632247 15557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 277a7093c46843c4bb8fe76c65323260 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:34:01.634483 15564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 277a7093c46843c4bb8fe76c65323260 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 277a7093c46843c4bb8fe76c65323260. Latest consensus state: current_term: 1 leader_uuid: "277a7093c46843c4bb8fe76c65323260" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "277a7093c46843c4bb8fe76c65323260" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 38051 } } }
I20260929 20:34:01.634821 15564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 277a7093c46843c4bb8fe76c65323260 [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:01.634588 15563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 277a7093c46843c4bb8fe76c65323260 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "277a7093c46843c4bb8fe76c65323260" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "277a7093c46843c4bb8fe76c65323260" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 38051 } } }
I20260929 20:34:01.634969 15563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 277a7093c46843c4bb8fe76c65323260 [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:01.635644 15567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:34:01.642529 15567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:34:01.653569 15567 catalog_manager.cc:1347] Generated new cluster ID: 52c0e37f540240718c35c67b84b43d75
I20260929 20:34:01.653702 15567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:34:01.678745 15567 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:34:01.680279 15567 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:34:01.688299 15567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 277a7093c46843c4bb8fe76c65323260: Generated new TSK 0
I20260929 20:34:01.689459 15567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.244.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab.
Entry for principal kudu/127.28.244.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab.
I20260929 20:34:01.771481 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.129:0
--local_ip_for_outbound_sockets=127.28.244.129
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:38051
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:46821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:34:02.126910 15583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:34:02.127310 15583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:34:02.127485 15583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:34:02.135531 15583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:34:02.135700 15583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:34:02.135905 15583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.129
I20260929 20:34:02.147765 15583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:46821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:38051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.129
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:34:02.149722 15583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:34:02.151628 15583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:34:02.165942 15589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:02.165900 15591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:02.165967 15588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:34:02.166734 15583 server_base.cc:1034] running on GCE node
I20260929 20:34:02.167414 15583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:34:02.168521 15583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:34:02.169725 15583 hybrid_clock.cc:648] HybridClock initialized: now 1790714042169657 us; error 62 us; skew 500 ppm
I20260929 20:34:02.170071 15583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:34:02 dist-test-slave-p29l krb5kdc[15476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714042, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:34:02.178678 15583 init.cc:377] Logged in from keytab as kudu/127.28.244.129@KRBTEST.COM (short username kudu)
I20260929 20:34:02.180426 15583 webserver.cc:460] Webserver started at http://127.28.244.129:40789/ using document root <none> and password file <none>
I20260929 20:34:02.181143 15583 fs_manager.cc:359] Metadata directory not provided
I20260929 20:34:02.181257 15583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:34:02.181543 15583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:34:02.184669 15583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-0/data/instance:
uuid: "45c8b407339d4cac817d07a361f5f3f4"
format_stamp: "Formatted at 2026-09-29 20:34:02 on dist-test-slave-p29l"
server_key: "0101b7ff9eeb5aa9d2b31e8b2dea4512"
server_key_iv: "9b1f2d79b025fb5c2878f20e6cd46af8"
server_key_version: "encryptionkey@0"
I20260929 20:34:02.185441 15583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance:
uuid: "45c8b407339d4cac817d07a361f5f3f4"
format_stamp: "Formatted at 2026-09-29 20:34:02 on dist-test-slave-p29l"
server_key: "0101b7ff9eeb5aa9d2b31e8b2dea4512"
server_key_iv: "9b1f2d79b025fb5c2878f20e6cd46af8"
server_key_version: "encryptionkey@0"
I20260929 20:34:02.191277 15583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 20:34:02.195237 15598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:02.197036 15583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:34:02.197190 15583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-0/wal
uuid: "45c8b407339d4cac817d07a361f5f3f4"
format_stamp: "Formatted at 2026-09-29 20:34:02 on dist-test-slave-p29l"
server_key: "0101b7ff9eeb5aa9d2b31e8b2dea4512"
server_key_iv: "9b1f2d79b025fb5c2878f20e6cd46af8"
server_key_version: "encryptionkey@0"
I20260929 20:34:02.197391 15583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:34:02.220952 15583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:34:02.225708 15583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:34:02.225966 15583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:34:02.227036 15583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:34:02.228601 15583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:34:02.228681 15583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:02.228770 15583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:34:02.228813 15583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:02.264539 15583 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.129:38307
I20260929 20:34:02.264650 15711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.129:38307 every 8 connection(s)
I20260929 20:34:02.265978 15583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
I20260929 20:34:02.271373 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 15583
I20260929 20:34:02.271541 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance
I20260929 20:34:02.272138 29650 external_mini_cluster.cc:1428] Setting key 2b2b9dd5b4c17083f89934a107c06f38
Sep 29 20:34:02 dist-test-slave-p29l krb5kdc[15476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714042, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:34:02.288331 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:02.268975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:34535 (local address 127.28.244.190:38051)
0929 20:34:02.269474 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:02.269503 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:02.270618 (+ 1115us) server_negotiation.cc:407] Connection header received
0929 20:34:02.272913 (+ 2295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:02.272923 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:02.273021 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:02.273115 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:02.274948 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:02.275704 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:02.277395 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:02.277829 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:02.277978 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:02.281942 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:02.281948 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:02.281951 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:02.281991 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:02.284849 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:02.285739 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:02.285744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:02.285746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:02.285881 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:02.286382 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:02.286387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:02.286389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:02.286676 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:02.286841 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:02.287651 (+ 810us) server_negotiation.cc:299] Negotiation successful
0929 20:34:02.287947 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":231,"threads_started":1}
I20260929 20:34:02.289408 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:02.269678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:38051 (local address 127.28.244.129:34535)
0929 20:34:02.270429 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:02.270477 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:02.272567 (+ 2090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:02.273413 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:02.273422 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:02.274132 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:34:02.274750 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:02.274775 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:02.275918 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:02.275922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:02.277217 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:02.277231 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:02.278081 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:02.278090 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:02.278367 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:02.279183 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:34:02.279210 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:34:02.281652 (+ 2442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:34:02.285039 (+ 3387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:02.285050 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:02.285079 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:02.285575 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:02.286004 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:02.286009 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:02.286013 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:02.286238 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:02.286822 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:34:02.286832 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:34:02.287232 (+ 400us) client_negotiation.cc:769] Sending connection context
0929 20:34:02.287523 (+ 291us) client_negotiation.cc:240] Negotiation successful
0929 20:34:02.288171 (+ 648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":278,"threads_started":1}
I20260929 20:34:02.291530 15712 heartbeater.cc:347] Connected to a master server at 127.28.244.190:38051
I20260929 20:34:02.292043 15712 heartbeater.cc:464] Registering TS with master...
I20260929 20:34:02.292929 15712 heartbeater.cc:511] Master 127.28.244.190:38051 requested a full tablet report, sending...
I20260929 20:34:02.295778 15521 ts_manager.cc:194] Registered new tserver with Master: 45c8b407339d4cac817d07a361f5f3f4 (127.28.244.129:38307)
WARNING: no policy specified for kudu/127.28.244.130@KRBTEST.COM; defaulting to no policy
I20260929 20:34:02.298158 15521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.129@KRBTEST.COM'} at 127.28.244.129:34535
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab.
Entry for principal kudu/127.28.244.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab.
I20260929 20:34:02.331669 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.130:0
--local_ip_for_outbound_sockets=127.28.244.130
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:38051
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:46821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:34:02.689603 15718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:34:02.689920 15718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:34:02.690023 15718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:34:02.698386 15718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:34:02.698519 15718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:34:02.698638 15718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.130
I20260929 20:34:02.709028 15718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:46821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:38051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.130
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:34:02.710371 15718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:34:02.711699 15718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:34:02.726179 15726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:02.726320 15724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:02.726366 15723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:34:02.729600 15718 server_base.cc:1034] running on GCE node
I20260929 20:34:02.730403 15718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:34:02.731496 15718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:34:02.732757 15718 hybrid_clock.cc:648] HybridClock initialized: now 1790714042732672 us; error 65 us; skew 500 ppm
I20260929 20:34:02.733139 15718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:34:02 dist-test-slave-p29l krb5kdc[15476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714042, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:34:02.736631 15718 init.cc:377] Logged in from keytab as kudu/127.28.244.130@KRBTEST.COM (short username kudu)
I20260929 20:34:02.738579 15718 webserver.cc:460] Webserver started at http://127.28.244.130:42245/ using document root <none> and password file <none>
I20260929 20:34:02.739380 15718 fs_manager.cc:359] Metadata directory not provided
I20260929 20:34:02.739534 15718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:34:02.739965 15718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:34:02.743103 15718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-1/data/instance:
uuid: "a14ff6c94cee4effbbedcd97aae4aec4"
format_stamp: "Formatted at 2026-09-29 20:34:02 on dist-test-slave-p29l"
server_key: "49f182265456812d558171600591c3bb"
server_key_iv: "24a31ed75422dd93e2cc87936f7185a6"
server_key_version: "encryptionkey@0"
I20260929 20:34:02.744059 15718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance:
uuid: "a14ff6c94cee4effbbedcd97aae4aec4"
format_stamp: "Formatted at 2026-09-29 20:34:02 on dist-test-slave-p29l"
server_key: "49f182265456812d558171600591c3bb"
server_key_iv: "24a31ed75422dd93e2cc87936f7185a6"
server_key_version: "encryptionkey@0"
I20260929 20:34:02.749779 15718 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:34:02.753757 15733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:02.755321 15718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:34:02.755523 15718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-1/wal
uuid: "a14ff6c94cee4effbbedcd97aae4aec4"
format_stamp: "Formatted at 2026-09-29 20:34:02 on dist-test-slave-p29l"
server_key: "49f182265456812d558171600591c3bb"
server_key_iv: "24a31ed75422dd93e2cc87936f7185a6"
server_key_version: "encryptionkey@0"
I20260929 20:34:02.755754 15718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:34:02.774652 15718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:34:02.779374 15718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:34:02.779691 15718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:34:02.780751 15718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:34:02.782260 15718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:34:02.782377 15718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:02.782538 15718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:34:02.782610 15718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:02.816982 15718 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.130:40179
I20260929 20:34:02.817040 15846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.130:40179 every 8 connection(s)
I20260929 20:34:02.818393 15718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
I20260929 20:34:02.819463 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 15718
I20260929 20:34:02.819671 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance
I20260929 20:34:02.820422 29650 external_mini_cluster.cc:1428] Setting key 63dba80c7e7cab077fab5b4a2fbbe991
Sep 29 20:34:02 dist-test-slave-p29l krb5kdc[15476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714042, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:34:02.844391 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:02.821698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:42463 (local address 127.28.244.190:38051)
0929 20:34:02.822757 (+ 1059us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:02.822767 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:02.823580 (+ 813us) server_negotiation.cc:407] Connection header received
0929 20:34:02.825527 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:02.825536 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:02.825655 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:02.825805 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:02.827611 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:02.829001 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:02.830760 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:02.831614 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:02.831741 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:02.836281 (+ 4540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:02.836291 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:02.836298 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:02.836351 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:02.840384 (+ 4033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:02.841401 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:02.841410 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:02.841416 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:02.841593 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:02.842116 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:02.842124 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:02.842130 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:02.842556 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:02.842715 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:02.843548 (+ 833us) server_negotiation.cc:299] Negotiation successful
0929 20:34:02.843885 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":911,"thread_start_us":802,"threads_started":1}
I20260929 20:34:02.845335 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:02.822378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:38051 (local address 127.28.244.130:42463)
0929 20:34:02.823397 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:02.823445 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:02.825211 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:02.826067 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:02.826076 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:02.826731 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:34:02.827419 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:02.827438 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:02.829196 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:02.829202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:02.830573 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:02.830586 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:02.831918 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:02.831928 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:02.832294 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:02.833320 (+ 1026us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:34:02.833354 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:34:02.835958 (+ 2604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:34:02.840627 (+ 4669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:02.840637 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:02.840655 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:02.841243 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:02.841729 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:02.841733 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:02.841736 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:02.841983 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:02.842702 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:34:02.842710 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:34:02.843131 (+ 421us) client_negotiation.cc:769] Sending connection context
0929 20:34:02.843411 (+ 280us) client_negotiation.cc:240] Negotiation successful
0929 20:34:02.843969 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":467,"threads_started":1}
I20260929 20:34:02.847272 15847 heartbeater.cc:347] Connected to a master server at 127.28.244.190:38051
I20260929 20:34:02.847671 15847 heartbeater.cc:464] Registering TS with master...
I20260929 20:34:02.848591 15847 heartbeater.cc:511] Master 127.28.244.190:38051 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.244.131@KRBTEST.COM; defaulting to no policy
I20260929 20:34:02.850502 15521 ts_manager.cc:194] Registered new tserver with Master: a14ff6c94cee4effbbedcd97aae4aec4 (127.28.244.130:40179)
I20260929 20:34:02.851572 15521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.130@KRBTEST.COM'} at 127.28.244.130:42463
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab.
Entry for principal kudu/127.28.244.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab.
I20260929 20:34:02.883450 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.131:0
--local_ip_for_outbound_sockets=127.28.244.131
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:38051
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:46821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:34:03.229188 15853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:34:03.229503 15853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:34:03.229657 15853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:34:03.237385 15853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:34:03.237571 15853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:34:03.237757 15853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.131
I20260929 20:34:03.248282 15853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:46821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:38051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.131
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:34:03.249941 15853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:34:03.251586 15853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:34:03.267455 15858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:03.267426 15859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:03.268464 15861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:34:03.268942 15853 server_base.cc:1034] running on GCE node
I20260929 20:34:03.269663 15853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:34:03.270821 15853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:34:03.272078 15853 hybrid_clock.cc:648] HybridClock initialized: now 1790714043272007 us; error 61 us; skew 500 ppm
I20260929 20:34:03.272501 15853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:34:03 dist-test-slave-p29l krb5kdc[15476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714043, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:34:03.276105 15853 init.cc:377] Logged in from keytab as kudu/127.28.244.131@KRBTEST.COM (short username kudu)
I20260929 20:34:03.277940 15853 webserver.cc:460] Webserver started at http://127.28.244.131:38643/ using document root <none> and password file <none>
I20260929 20:34:03.278831 15853 fs_manager.cc:359] Metadata directory not provided
I20260929 20:34:03.278991 15853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:34:03.279340 15853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:34:03.282723 15853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-2/data/instance:
uuid: "40e66d125d294ad6a81ad404db6c1388"
format_stamp: "Formatted at 2026-09-29 20:34:03 on dist-test-slave-p29l"
server_key: "8a8882e4229714e6441fd9e9e95b142b"
server_key_iv: "277b29ceca518100d8e47a68a44c30cf"
server_key_version: "encryptionkey@0"
I20260929 20:34:03.283638 15853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance:
uuid: "40e66d125d294ad6a81ad404db6c1388"
format_stamp: "Formatted at 2026-09-29 20:34:03 on dist-test-slave-p29l"
server_key: "8a8882e4229714e6441fd9e9e95b142b"
server_key_iv: "277b29ceca518100d8e47a68a44c30cf"
server_key_version: "encryptionkey@0"
I20260929 20:34:03.289980 15853 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 20:34:03.294381 15868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:03.296177 15853 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 20:34:03.296413 15853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-2/wal
uuid: "40e66d125d294ad6a81ad404db6c1388"
format_stamp: "Formatted at 2026-09-29 20:34:03 on dist-test-slave-p29l"
server_key: "8a8882e4229714e6441fd9e9e95b142b"
server_key_iv: "277b29ceca518100d8e47a68a44c30cf"
server_key_version: "encryptionkey@0"
I20260929 20:34:03.296667 15853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:34:03.302100 15712 heartbeater.cc:503] Master 127.28.244.190:38051 was elected leader, sending a full tablet report...
I20260929 20:34:03.318830 15853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:34:03.323668 15853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:34:03.324023 15853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:34:03.325062 15853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:34:03.326646 15853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:34:03.326751 15853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:03.326912 15853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:34:03.326977 15853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:03.362658 15853 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.131:34631
I20260929 20:34:03.362783 15980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.131:34631 every 8 connection(s)
I20260929 20:34:03.364130 15853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
I20260929 20:34:03.370428 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 15853
I20260929 20:34:03.370616 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance
I20260929 20:34:03.371187 29650 external_mini_cluster.cc:1428] Setting key a0a2a8ce08bd3ecc6e35f3c3c3713e01
Sep 29 20:34:03 dist-test-slave-p29l krb5kdc[15476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714043, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:34:03.385587 15983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:03.366915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:42953 (local address 127.28.244.190:38051)
0929 20:34:03.367427 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:03.367434 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:03.368565 (+ 1131us) server_negotiation.cc:407] Connection header received
0929 20:34:03.370492 (+ 1927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:03.370500 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:03.370614 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:03.370748 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:03.372855 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:03.373850 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:03.375609 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:03.376071 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:03.376172 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:03.379671 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:03.379676 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:03.379679 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:03.379717 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:03.382396 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:03.383173 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:03.383176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:03.383179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:03.383304 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:03.383710 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:03.383713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:03.383715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:03.384065 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:03.384188 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:03.384957 (+ 769us) server_negotiation.cc:299] Negotiation successful
0929 20:34:03.385162 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":233,"threads_started":1}
I20260929 20:34:03.386855 15984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:03.367533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:38051 (local address 127.28.244.131:42953)
0929 20:34:03.368378 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:03.368425 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:03.370169 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:03.371036 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:03.371046 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:03.371698 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:34:03.372634 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:03.372665 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:03.374027 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:03.374034 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:03.375464 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:03.375479 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:03.376268 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:03.376273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:03.376483 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:03.377143 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:34:03.377171 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:34:03.379425 (+ 2254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:34:03.382558 (+ 3133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:03.382566 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:03.382581 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:03.383055 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:03.383422 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:03.383425 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:03.383427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:03.383614 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:03.384180 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:34:03.384186 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:34:03.384553 (+ 367us) client_negotiation.cc:769] Sending connection context
0929 20:34:03.384799 (+ 246us) client_negotiation.cc:240] Negotiation successful
0929 20:34:03.385353 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":392,"threads_started":1}
I20260929 20:34:03.388717 15981 heartbeater.cc:347] Connected to a master server at 127.28.244.190:38051
I20260929 20:34:03.389077 15981 heartbeater.cc:464] Registering TS with master...
I20260929 20:34:03.389849 15981 heartbeater.cc:511] Master 127.28.244.190:38051 requested a full tablet report, sending...
I20260929 20:34:03.391440 15521 ts_manager.cc:194] Registered new tserver with Master: 40e66d125d294ad6a81ad404db6c1388 (127.28.244.131:34631)
I20260929 20:34:03.392347 15521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.131@KRBTEST.COM'} at 127.28.244.131:42953
I20260929 20:34:03.400162 29650 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:34:03.418470 15983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:03.404609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52576 (local address 127.28.244.190:38051)
0929 20:34:03.404854 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:03.404865 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:03.405300 (+ 435us) server_negotiation.cc:407] Connection header received
0929 20:34:03.405448 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:03.405452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:03.405522 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:03.405606 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:03.407063 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:03.407896 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:03.409313 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:03.409731 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:03.409848 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:03.412653 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:03.412658 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:03.412661 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:03.412701 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:03.415295 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:03.416284 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:03.416289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:03.416291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:03.416414 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:03.416888 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:03.416891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:03.416893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:03.417228 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:03.417419 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:03.417852 (+ 433us) server_negotiation.cc:299] Negotiation successful
0929 20:34:03.418000 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 20:34:03.422956 15521 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:34:03.430347 15521 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:34:03.465197 15994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:03.444042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:40179 (local address 127.0.0.1:57228)
0929 20:34:03.452163 (+ 8121us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:03.452206 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:03.452422 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:03.454090 (+ 1668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:03.454103 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:03.454152 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:34:03.454759 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:03.454775 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:03.457746 (+ 2971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:03.457755 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:03.460890 (+ 3135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:03.460916 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:03.463712 (+ 2796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:03.463723 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:03.464066 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:03.464157 (+ 91us) client_negotiation.cc:769] Sending connection context
0929 20:34:03.464339 (+ 182us) client_negotiation.cc:240] Negotiation successful
0929 20:34:03.464644 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7840,"thread_start_us":7665,"threads_started":1}
I20260929 20:34:03.467648 15995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:03.444243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57228 (local address 127.28.244.130:40179)
0929 20:34:03.453215 (+ 8972us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:03.453230 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:03.453269 (+ 39us) server_negotiation.cc:407] Connection header received
0929 20:34:03.453418 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:03.453430 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:03.453767 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:03.455047 (+ 1280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:34:03.455360 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:03.457515 (+ 2155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:03.461195 (+ 3680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:03.463368 (+ 2173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:03.466463 (+ 3095us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:03.466771 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:03.466874 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 20:34:03.467134 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8753,"thread_start_us":8574,"threads_started":1}
I20260929 20:34:03.473016 15781 tablet_service.cc:1467] Processing CreateTablet for tablet 97ba547488f74ddba2b9cb7e8fc33d52 (DEFAULT_TABLE table=test-table [id=452ad2a0af66434f88376f92e9f4c19c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:34:03.474676 15781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97ba547488f74ddba2b9cb7e8fc33d52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:34:03.479785 15994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:03.445093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:38307 (local address 127.0.0.1:59124)
0929 20:34:03.465607 (+ 20514us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:03.465637 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:03.465787 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:03.473370 (+ 7583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:03.473375 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:03.473409 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:34:03.473595 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:03.473608 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:03.475558 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:03.475564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:03.477158 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:03.477173 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:03.478883 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:03.478889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:03.479094 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:03.479134 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 20:34:03.479212 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 20:34:03.479401 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20356,"mutex_wait_us":1,"thread_start_us":24842,"threads_started":1}
I20260929 20:34:03.481077 15996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:03.444749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59124 (local address 127.28.244.129:38307)
0929 20:34:03.472599 (+ 27850us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:03.472612 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:03.472649 (+ 37us) server_negotiation.cc:407] Connection header received
0929 20:34:03.472805 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:03.472815 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:03.473106 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:03.473373 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:34:03.473798 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:03.475381 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:03.477348 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:03.478665 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:03.480193 (+ 1528us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:03.480416 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:03.480505 (+ 89us) server_negotiation.cc:299] Negotiation successful
0929 20:34:03.480706 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27655,"thread_start_us":27477,"threads_started":1}
I20260929 20:34:03.481125 15998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:03.445487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:34631 (local address 127.0.0.1:49600)
0929 20:34:03.469809 (+ 24322us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:03.469835 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:03.469982 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:03.473683 (+ 3701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:03.473691 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:03.473724 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:34:03.474172 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:03.474185 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:03.475914 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:03.475919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:03.478126 (+ 2207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:03.478146 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:03.480292 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:03.480300 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:03.480490 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:03.480531 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 20:34:03.480639 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 20:34:03.480775 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24145,"thread_start_us":23962,"threads_started":1}
I20260929 20:34:03.481230 15999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:03.445834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49600 (local address 127.28.244.131:34631)
0929 20:34:03.473038 (+ 27204us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:03.473047 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:03.473082 (+ 35us) server_negotiation.cc:407] Connection header received
0929 20:34:03.473187 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:03.473194 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:03.473422 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:03.473625 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:34:03.474398 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:03.475709 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:03.478334 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:03.479775 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:03.479983 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:03.480171 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:03.480701 (+ 530us) server_negotiation.cc:299] Negotiation successful
0929 20:34:03.480878 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27019,"thread_start_us":26826,"threads_started":1}
I20260929 20:34:03.485649 15916 tablet_service.cc:1467] Processing CreateTablet for tablet 97ba547488f74ddba2b9cb7e8fc33d52 (DEFAULT_TABLE table=test-table [id=452ad2a0af66434f88376f92e9f4c19c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:34:03.485649 15646 tablet_service.cc:1467] Processing CreateTablet for tablet 97ba547488f74ddba2b9cb7e8fc33d52 (DEFAULT_TABLE table=test-table [id=452ad2a0af66434f88376f92e9f4c19c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:34:03.487468 15916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97ba547488f74ddba2b9cb7e8fc33d52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:34:03.487465 15646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97ba547488f74ddba2b9cb7e8fc33d52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:34:03.488457 16000 tablet_bootstrap.cc:492] T 97ba547488f74ddba2b9cb7e8fc33d52 P a14ff6c94cee4effbbedcd97aae4aec4: Bootstrap starting.
I20260929 20:34:03.491647 16000 tablet_bootstrap.cc:654] T 97ba547488f74ddba2b9cb7e8fc33d52 P a14ff6c94cee4effbbedcd97aae4aec4: Neither blocks nor log segments found. Creating new log.
I20260929 20:34:03.492749 16000 log.cc:824] T 97ba547488f74ddba2b9cb7e8fc33d52 P a14ff6c94cee4effbbedcd97aae4aec4: Log is configured to *not* fsync() on all Append() calls
I20260929 20:34:03.495636 16000 tablet_bootstrap.cc:492] T 97ba547488f74ddba2b9cb7e8fc33d52 P a14ff6c94cee4effbbedcd97aae4aec4: No bootstrap required, opened a new log
I20260929 20:34:03.495975 16000 ts_tablet_manager.cc:1397] T 97ba547488f74ddba2b9cb7e8fc33d52 P a14ff6c94cee4effbbedcd97aae4aec4: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260929 20:34:03.504860 16002 tablet_bootstrap.cc:492] T 97ba547488f74ddba2b9cb7e8fc33d52 P 45c8b407339d4cac817d07a361f5f3f4: Bootstrap starting.
I20260929 20:34:03.504686 16000 raft_consensus.cc:348] T 97ba547488f74ddba2b9cb7e8fc33d52 P a14ff6c94cee4effbbedcd97aae4aec4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a14ff6c94cee4effbbedcd97aae4aec4" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40179 } } peers { permanent_uuid: "45c8b407339d4cac817d07a361f5f3f4" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38307 } } peers { permanent_uuid: "40e66d125d294ad6a81ad404db6c1388" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 34631 } }
I20260929 20:34:03.505187 16000 raft_consensus.cc:374] T 97ba547488f74ddba2b9cb7e8fc33d52 P a14ff6c94cee4effbbedcd97aae4aec4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:34:03.505308 16000 raft_consensus.cc:729] T 97ba547488f74ddba2b9cb7e8fc33d52 P a14ff6c94cee4effbbedcd97aae4aec4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a14ff6c94cee4effbbedcd97aae4aec4, State: Initialized, Role: FOLLOWER
I20260929 20:34:03.506083 16000 consensus_queue.cc:260] T 97ba547488f74ddba2b9cb7e8fc33d52 P a14ff6c94cee4effbbedcd97aae4aec4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a14ff6c94cee4effbbedcd97aae4aec4" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40179 } } peers { permanent_uuid: "45c8b407339d4cac817d07a361f5f3f4" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38307 } } peers { permanent_uuid: "40e66d125d294ad6a81ad404db6c1388" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 34631 } }
I20260929 20:34:03.507472 15847 heartbeater.cc:503] Master 127.28.244.190:38051 was elected leader, sending a full tablet report...
I20260929 20:34:03.509251 16002 tablet_bootstrap.cc:654] T 97ba547488f74ddba2b9cb7e8fc33d52 P 45c8b407339d4cac817d07a361f5f3f4: Neither blocks nor log segments found. Creating new log.
I20260929 20:34:03.509626 16000 ts_tablet_manager.cc:1428] T 97ba547488f74ddba2b9cb7e8fc33d52 P a14ff6c94cee4effbbedcd97aae4aec4: Time spent starting tablet: real 0.014s user 0.007s sys 0.005s
I20260929 20:34:03.510701 16002 log.cc:824] T 97ba547488f74ddba2b9cb7e8fc33d52 P 45c8b407339d4cac817d07a361f5f3f4: Log is configured to *not* fsync() on all Append() calls
I20260929 20:34:03.510761 16003 tablet_bootstrap.cc:492] T 97ba547488f74ddba2b9cb7e8fc33d52 P 40e66d125d294ad6a81ad404db6c1388: Bootstrap starting.
I20260929 20:34:03.515447 16002 tablet_bootstrap.cc:492] T 97ba547488f74ddba2b9cb7e8fc33d52 P 45c8b407339d4cac817d07a361f5f3f4: No bootstrap required, opened a new log
I20260929 20:34:03.515789 16002 ts_tablet_manager.cc:1397] T 97ba547488f74ddba2b9cb7e8fc33d52 P 45c8b407339d4cac817d07a361f5f3f4: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260929 20:34:03.515875 16003 tablet_bootstrap.cc:654] T 97ba547488f74ddba2b9cb7e8fc33d52 P 40e66d125d294ad6a81ad404db6c1388: Neither blocks nor log segments found. Creating new log.
I20260929 20:34:03.517308 16003 log.cc:824] T 97ba547488f74ddba2b9cb7e8fc33d52 P 40e66d125d294ad6a81ad404db6c1388: Log is configured to *not* fsync() on all Append() calls
I20260929 20:34:03.520944 16003 tablet_bootstrap.cc:492] T 97ba547488f74ddba2b9cb7e8fc33d52 P 40e66d125d294ad6a81ad404db6c1388: No bootstrap required, opened a new log
I20260929 20:34:03.521270 16003 ts_tablet_manager.cc:1397] T 97ba547488f74ddba2b9cb7e8fc33d52 P 40e66d125d294ad6a81ad404db6c1388: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260929 20:34:03.527945 16002 raft_consensus.cc:348] T 97ba547488f74ddba2b9cb7e8fc33d52 P 45c8b407339d4cac817d07a361f5f3f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a14ff6c94cee4effbbedcd97aae4aec4" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40179 } } peers { permanent_uuid: "45c8b407339d4cac817d07a361f5f3f4" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38307 } } peers { permanent_uuid: "40e66d125d294ad6a81ad404db6c1388" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 34631 } }
I20260929 20:34:03.528487 16002 raft_consensus.cc:374] T 97ba547488f74ddba2b9cb7e8fc33d52 P 45c8b407339d4cac817d07a361f5f3f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:34:03.528604 16002 raft_consensus.cc:729] T 97ba547488f74ddba2b9cb7e8fc33d52 P 45c8b407339d4cac817d07a361f5f3f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45c8b407339d4cac817d07a361f5f3f4, State: Initialized, Role: FOLLOWER
I20260929 20:34:03.529361 16002 consensus_queue.cc:260] T 97ba547488f74ddba2b9cb7e8fc33d52 P 45c8b407339d4cac817d07a361f5f3f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a14ff6c94cee4effbbedcd97aae4aec4" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40179 } } peers { permanent_uuid: "45c8b407339d4cac817d07a361f5f3f4" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38307 } } peers { permanent_uuid: "40e66d125d294ad6a81ad404db6c1388" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 34631 } }
I20260929 20:34:03.531567 16002 ts_tablet_manager.cc:1428] T 97ba547488f74ddba2b9cb7e8fc33d52 P 45c8b407339d4cac817d07a361f5f3f4: Time spent starting tablet: real 0.016s user 0.013s sys 0.001s
I20260929 20:34:03.533043 16003 raft_consensus.cc:348] T 97ba547488f74ddba2b9cb7e8fc33d52 P 40e66d125d294ad6a81ad404db6c1388 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a14ff6c94cee4effbbedcd97aae4aec4" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40179 } } peers { permanent_uuid: "45c8b407339d4cac817d07a361f5f3f4" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38307 } } peers { permanent_uuid: "40e66d125d294ad6a81ad404db6c1388" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 34631 } }
I20260929 20:34:03.533491 16003 raft_consensus.cc:374] T 97ba547488f74ddba2b9cb7e8fc33d52 P 40e66d125d294ad6a81ad404db6c1388 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:34:03.533612 16003 raft_consensus.cc:729] T 97ba547488f74ddba2b9cb7e8fc33d52 P 40e66d125d294ad6a81ad404db6c1388 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40e66d125d294ad6a81ad404db6c1388, State: Initialized, Role: FOLLOWER
I20260929 20:34:03.534225 16003 consensus_queue.cc:260] T 97ba547488f74ddba2b9cb7e8fc33d52 P 40e66d125d294ad6a81ad404db6c1388 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a14ff6c94cee4effbbedcd97aae4aec4" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40179 } } peers { permanent_uuid: "45c8b407339d4cac817d07a361f5f3f4" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38307 } } peers { permanent_uuid: "40e66d125d294ad6a81ad404db6c1388" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 34631 } }
I20260929 20:34:03.535349 15981 heartbeater.cc:503] Master 127.28.244.190:38051 was elected leader, sending a full tablet report...
I20260929 20:34:03.536329 16003 ts_tablet_manager.cc:1428] T 97ba547488f74ddba2b9cb7e8fc33d52 P 40e66d125d294ad6a81ad404db6c1388: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
W20260929 20:34:03.571184 15848 tablet.cc:2369] T 97ba547488f74ddba2b9cb7e8fc33d52 P a14ff6c94cee4effbbedcd97aae4aec4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:34:03.593617 16004 raft_consensus.cc:482] T 97ba547488f74ddba2b9cb7e8fc33d52 P a14ff6c94cee4effbbedcd97aae4aec4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:34:03.593747 16007 raft_consensus.cc:482] T 97ba547488f74ddba2b9cb7e8fc33d52 P 45c8b407339d4cac817d07a361f5f3f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:34:03.594010 16004 raft_consensus.cc:504] T 97ba547488f74ddba2b9cb7e8fc33d52 P a14ff6c94cee4effbbedcd97aae4aec4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a14ff6c94cee4effbbedcd97aae4aec4" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40179 } } peers { permanent_uuid: "45c8b407339d4cac817d07a361f5f3f4" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38307 } } peers { permanent_uuid: "40e66d125d294ad6a81ad404db6c1388" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 34631 } }
I20260929 20:34:03.594065 16007 raft_consensus.cc:504] T 97ba547488f74ddba2b9cb7e8fc33d52 P 45c8b407339d4cac817d07a361f5f3f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a14ff6c94cee4effbbedcd97aae4aec4" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40179 } } peers { permanent_uuid: "45c8b407339d4cac817d07a361f5f3f4" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38307 } } peers { permanent_uuid: "40e66d125d294ad6a81ad404db6c1388" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 34631 } }
I20260929 20:34:03.595943 16004 leader_election.cc:288] T 97ba547488f74ddba2b9cb7e8fc33d52 P a14ff6c94cee4effbbedcd97aae4aec4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45c8b407339d4cac817d07a361f5f3f4 (127.28.244.129:38307), 40e66d125d294ad6a81ad404db6c1388 (127.28.244.131:34631)
I20260929 20:34:03.596155 16007 leader_election.cc:288] T 97ba547488f74ddba2b9cb7e8fc33d52 P 45c8b407339d4cac817d07a361f5f3f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a14ff6c94cee4effbbedcd97aae4aec4 (127.28.244.130:40179), 40e66d125d294ad6a81ad404db6c1388 (127.28.244.131:34631)
W20260929 20:34:03.615836 15982 tablet.cc:2369] T 97ba547488f74ddba2b9cb7e8fc33d52 P 40e66d125d294ad6a81ad404db6c1388: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:34:03.618973 15995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:03.610171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:47145 (local address 127.28.244.130:40179)
0929 20:34:03.610363 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:03.610368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:03.610387 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:34:03.610445 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:03.610450 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:03.610524 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:03.610617 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:34:03.611839 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:03.613269 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:03.615998 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:03.617580 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:03.617772 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:03.617863 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:03.618485 (+ 622us) server_negotiation.cc:299] Negotiation successful
0929 20:34:03.618641 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":139,"server-negotiator.queue_time_us":56}
I20260929 20:34:03.619594 16009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:03.596433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:40179 (local address 127.28.244.129:47145)
0929 20:34:03.609125 (+ 12692us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:03.609163 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:03.609350 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:03.611174 (+ 1824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:03.611182 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:03.611218 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:34:03.611652 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:03.611664 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:03.613437 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:03.613443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:03.615796 (+ 2353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:03.615833 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:03.618024 (+ 2191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:03.618031 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:03.618250 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:03.618297 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 20:34:03.618976 (+ 679us) client_negotiation.cc:240] Negotiation successful
0929 20:34:03.619149 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12479,"thread_start_us":13019,"threads_started":1}
I20260929 20:34:03.620440 15999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:03.596780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:32927 (local address 127.28.244.131:34631)
0929 20:34:03.597065 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:03.597074 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:03.610767 (+ 13693us) server_negotiation.cc:407] Connection header received
0929 20:34:03.610875 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:03.610886 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:03.610963 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:03.611064 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:34:03.613160 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:03.614342 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:03.617734 (+ 3392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:03.619018 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:03.619166 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:03.619233 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:03.619972 (+ 739us) server_negotiation.cc:299] Negotiation successful
0929 20:34:03.620127 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 20:34:03.620636 16011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:03.596319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:34631 (local address 127.28.244.130:32927)
0929 20:34:03.610223 (+ 13904us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:03.610242 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:03.610374 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:03.611304 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:03.611311 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:03.611343 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:34:03.612978 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:03.612989 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:03.614541 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:03.614546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:03.617524 (+ 2978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:03.617545 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:03.619571 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:03.619577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:03.619758 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:03.619791 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 20:34:03.619911 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 20:34:03.620058 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13774,"mutex_wait_us":98,"thread_start_us":13473,"threads_started":1}
I20260929 20:34:03.620769 15801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ba547488f74ddba2b9cb7e8fc33d52" candidate_uuid: "45c8b407339d4cac817d07a361f5f3f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a14ff6c94cee4effbbedcd97aae4aec4" is_pre_election: true
I20260929 20:34:03.621172 16010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:03.596307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:38307 (local address 127.28.244.130:52279)
0929 20:34:03.609756 (+ 13449us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:03.609788 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:03.610011 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:03.611874 (+ 1863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:03.611880 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:03.611907 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:34:03.612445 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:03.612486 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:03.614273 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:03.614282 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:03.618040 (+ 3758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:03.618055 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:03.620023 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:03.620030 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:03.620216 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:03.620301 (+ 85us) client_negotiation.cc:769] Sending connection context
0929 20:34:03.620405 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 20:34:03.620581 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13207,"mutex_wait_us":766,"thread_start_us":12939,"threads_started":1}
I20260929 20:34:03.621325 15801 raft_consensus.cc:2454] T 97ba547488f74ddba2b9cb7e8fc33d52 P a14ff6c94cee4effbbedcd97aae4aec4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45c8b407339d4cac817d07a361f5f3f4 in term 0.
I20260929 20:34:03.621609 16012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:03.597063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:34631 (local address 127.28.244.129:37811)
0929 20:34:03.611309 (+ 14246us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:03.611327 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:03.611488 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:03.611980 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:03.611986 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:03.612010 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:34:03.612377 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:03.612392 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:03.614254 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:03.614259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:03.616572 (+ 2313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:03.616588 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:03.618756 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:03.618764 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:03.620837 (+ 2073us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:03.620875 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 20:34:03.620973 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 20:34:03.621159 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14098,"mutex_wait_us":1788,"thread_start_us":13922,"threads_started":1}
I20260929 20:34:03.621958 15996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:03.611299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:52279 (local address 127.28.244.129:38307)
0929 20:34:03.611516 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:03.611525 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:03.611544 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:34:03.611605 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:03.611611 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:03.611677 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:03.611761 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:34:03.612679 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:03.614042 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:03.618222 (+ 4180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:03.619554 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:03.619709 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:03.619781 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:03.621551 (+ 1770us) server_negotiation.cc:299] Negotiation successful
0929 20:34:03.621695 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":62}
I20260929 20:34:03.622113 15600 leader_election.cc:302] T 97ba547488f74ddba2b9cb7e8fc33d52 P 45c8b407339d4cac817d07a361f5f3f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45c8b407339d4cac817d07a361f5f3f4, a14ff6c94cee4effbbedcd97aae4aec4; no voters:
I20260929 20:34:03.622483 16013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:03.597037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:37811 (local address 127.28.244.131:34631)
0929 20:34:03.611042 (+ 14005us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:03.611049 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:03.611437 (+ 388us) server_negotiation.cc:407] Connection header received
0929 20:34:03.611667 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:03.611675 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:03.611793 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:03.611923 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:34:03.612617 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:03.614065 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:03.616787 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:03.618306 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:03.618494 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:03.618572 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:03.621915 (+ 3343us) server_negotiation.cc:299] Negotiation successful
0929 20:34:03.622106 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13848,"thread_start_us":13574,"threads_started":1}
I20260929 20:34:03.622617 16007 raft_consensus.cc:2781] T 97ba547488f74ddba2b9cb7e8fc33d52 P 45c8b407339d4cac817d07a361f5f3f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:34:03.622784 16007 raft_consensus.cc:482] T 97ba547488f74ddba2b9cb7e8fc33d52 P 45c8b407339d4cac817d07a361f5f3f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:34:03.622923 16007 raft_consensus.cc:3037] T 97ba547488f74ddba2b9cb7e8fc33d52 P 45c8b407339d4cac817d07a361f5f3f4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:34:03.622884 15666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ba547488f74ddba2b9cb7e8fc33d52" candidate_uuid: "a14ff6c94cee4effbbedcd97aae4aec4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45c8b407339d4cac817d07a361f5f3f4" is_pre_election: true
I20260929 20:34:03.623466 15936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ba547488f74ddba2b9cb7e8fc33d52" candidate_uuid: "a14ff6c94cee4effbbedcd97aae4aec4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40e66d125d294ad6a81ad404db6c1388" is_pre_election: true
I20260929 20:34:03.623539 15935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ba547488f74ddba2b9cb7e8fc33d52" candidate_uuid: "45c8b407339d4cac817d07a361f5f3f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40e66d125d294ad6a81ad404db6c1388" is_pre_election: true
I20260929 20:34:03.624154 15936 raft_consensus.cc:2454] T 97ba547488f74ddba2b9cb7e8fc33d52 P 40e66d125d294ad6a81ad404db6c1388 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a14ff6c94cee4effbbedcd97aae4aec4 in term 0.
I20260929 20:34:03.625456 15737 leader_election.cc:302] T 97ba547488f74ddba2b9cb7e8fc33d52 P a14ff6c94cee4effbbedcd97aae4aec4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40e66d125d294ad6a81ad404db6c1388, a14ff6c94cee4effbbedcd97aae4aec4; no voters:
I20260929 20:34:03.625958 16004 raft_consensus.cc:2781] T 97ba547488f74ddba2b9cb7e8fc33d52 P a14ff6c94cee4effbbedcd97aae4aec4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:34:03.626116 16004 raft_consensus.cc:482] T 97ba547488f74ddba2b9cb7e8fc33d52 P a14ff6c94cee4effbbedcd97aae4aec4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:34:03.626247 16004 raft_consensus.cc:3037] T 97ba547488f74ddba2b9cb7e8fc33d52 P a14ff6c94cee4effbbedcd97aae4aec4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:34:03.627043 16007 raft_consensus.cc:504] T 97ba547488f74ddba2b9cb7e8fc33d52 P 45c8b407339d4cac817d07a361f5f3f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a14ff6c94cee4effbbedcd97aae4aec4" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40179 } } peers { permanent_uuid: "45c8b407339d4cac817d07a361f5f3f4" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38307 } } peers { permanent_uuid: "40e66d125d294ad6a81ad404db6c1388" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 34631 } }
I20260929 20:34:03.627568 15666 raft_consensus.cc:2379] T 97ba547488f74ddba2b9cb7e8fc33d52 P 45c8b407339d4cac817d07a361f5f3f4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a14ff6c94cee4effbbedcd97aae4aec4 in current term 1: Already voted for candidate 45c8b407339d4cac817d07a361f5f3f4 in this term.
I20260929 20:34:03.628258 16007 leader_election.cc:288] T 97ba547488f74ddba2b9cb7e8fc33d52 P 45c8b407339d4cac817d07a361f5f3f4 [CANDIDATE]: Term 1 election: Requested vote from peers a14ff6c94cee4effbbedcd97aae4aec4 (127.28.244.130:40179), 40e66d125d294ad6a81ad404db6c1388 (127.28.244.131:34631)
I20260929 20:34:03.628865 15936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ba547488f74ddba2b9cb7e8fc33d52" candidate_uuid: "45c8b407339d4cac817d07a361f5f3f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40e66d125d294ad6a81ad404db6c1388"
I20260929 20:34:03.629122 15936 raft_consensus.cc:3037] T 97ba547488f74ddba2b9cb7e8fc33d52 P 40e66d125d294ad6a81ad404db6c1388 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:34:03.629354 16004 raft_consensus.cc:504] T 97ba547488f74ddba2b9cb7e8fc33d52 P a14ff6c94cee4effbbedcd97aae4aec4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a14ff6c94cee4effbbedcd97aae4aec4" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40179 } } peers { permanent_uuid: "45c8b407339d4cac817d07a361f5f3f4" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38307 } } peers { permanent_uuid: "40e66d125d294ad6a81ad404db6c1388" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 34631 } }
I20260929 20:34:03.630013 16004 leader_election.cc:288] T 97ba547488f74ddba2b9cb7e8fc33d52 P a14ff6c94cee4effbbedcd97aae4aec4 [CANDIDATE]: Term 1 election: Requested vote from peers 45c8b407339d4cac817d07a361f5f3f4 (127.28.244.129:38307), 40e66d125d294ad6a81ad404db6c1388 (127.28.244.131:34631)
I20260929 20:34:03.630458 15666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ba547488f74ddba2b9cb7e8fc33d52" candidate_uuid: "a14ff6c94cee4effbbedcd97aae4aec4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45c8b407339d4cac817d07a361f5f3f4"
I20260929 20:34:03.630721 15666 raft_consensus.cc:2379] T 97ba547488f74ddba2b9cb7e8fc33d52 P 45c8b407339d4cac817d07a361f5f3f4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a14ff6c94cee4effbbedcd97aae4aec4 in current term 1: Already voted for candidate 45c8b407339d4cac817d07a361f5f3f4 in this term.
I20260929 20:34:03.630888 15801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ba547488f74ddba2b9cb7e8fc33d52" candidate_uuid: "45c8b407339d4cac817d07a361f5f3f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a14ff6c94cee4effbbedcd97aae4aec4"
I20260929 20:34:03.631214 15801 raft_consensus.cc:2379] T 97ba547488f74ddba2b9cb7e8fc33d52 P a14ff6c94cee4effbbedcd97aae4aec4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 45c8b407339d4cac817d07a361f5f3f4 in current term 1: Already voted for candidate a14ff6c94cee4effbbedcd97aae4aec4 in this term.
I20260929 20:34:03.631165 15935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ba547488f74ddba2b9cb7e8fc33d52" candidate_uuid: "a14ff6c94cee4effbbedcd97aae4aec4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40e66d125d294ad6a81ad404db6c1388"
I20260929 20:34:03.632006 15737 leader_election.cc:302] T 97ba547488f74ddba2b9cb7e8fc33d52 P a14ff6c94cee4effbbedcd97aae4aec4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a14ff6c94cee4effbbedcd97aae4aec4; no voters: 40e66d125d294ad6a81ad404db6c1388, 45c8b407339d4cac817d07a361f5f3f4
I20260929 20:34:03.632314 16004 raft_consensus.cc:2726] T 97ba547488f74ddba2b9cb7e8fc33d52 P a14ff6c94cee4effbbedcd97aae4aec4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260929 20:34:03.632781 15936 raft_consensus.cc:2454] T 97ba547488f74ddba2b9cb7e8fc33d52 P 40e66d125d294ad6a81ad404db6c1388 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45c8b407339d4cac817d07a361f5f3f4 in term 1.
I20260929 20:34:03.633227 15602 leader_election.cc:302] T 97ba547488f74ddba2b9cb7e8fc33d52 P 45c8b407339d4cac817d07a361f5f3f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40e66d125d294ad6a81ad404db6c1388, 45c8b407339d4cac817d07a361f5f3f4; no voters: a14ff6c94cee4effbbedcd97aae4aec4
I20260929 20:34:03.633620 16007 raft_consensus.cc:2781] T 97ba547488f74ddba2b9cb7e8fc33d52 P 45c8b407339d4cac817d07a361f5f3f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:34:03.634122 16007 raft_consensus.cc:686] T 97ba547488f74ddba2b9cb7e8fc33d52 P 45c8b407339d4cac817d07a361f5f3f4 [term 1 LEADER]: Becoming Leader. State: Replica: 45c8b407339d4cac817d07a361f5f3f4, State: Running, Role: LEADER
I20260929 20:34:03.634625 16007 consensus_queue.cc:237] T 97ba547488f74ddba2b9cb7e8fc33d52 P 45c8b407339d4cac817d07a361f5f3f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a14ff6c94cee4effbbedcd97aae4aec4" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40179 } } peers { permanent_uuid: "45c8b407339d4cac817d07a361f5f3f4" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38307 } } peers { permanent_uuid: "40e66d125d294ad6a81ad404db6c1388" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 34631 } }
I20260929 20:34:03.639609 15521 catalog_manager.cc:5495] T 97ba547488f74ddba2b9cb7e8fc33d52 P 45c8b407339d4cac817d07a361f5f3f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 45c8b407339d4cac817d07a361f5f3f4 (127.28.244.129). New cstate: current_term: 1 leader_uuid: "45c8b407339d4cac817d07a361f5f3f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a14ff6c94cee4effbbedcd97aae4aec4" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45c8b407339d4cac817d07a361f5f3f4" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 38307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40e66d125d294ad6a81ad404db6c1388" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 34631 } health_report { overall_health: UNKNOWN } } }
I20260929 20:34:03.703905 15996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:03.696974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59134 (local address 127.28.244.129:38307)
0929 20:34:03.697221 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:03.697228 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:03.697253 (+ 25us) server_negotiation.cc:407] Connection header received
0929 20:34:03.697556 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:03.697565 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:03.697700 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:03.697906 (+ 206us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:34:03.698471 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:03.699312 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:03.701158 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:03.701656 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:03.701762 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:03.702549 (+ 787us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:34:03.703185 (+ 636us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:34:03.703350 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:03.703522 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 20:34:03.703568 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 20:34:03.712765 15801 raft_consensus.cc:1261] T 97ba547488f74ddba2b9cb7e8fc33d52 P a14ff6c94cee4effbbedcd97aae4aec4 [term 1 FOLLOWER]: Refusing update from remote peer 45c8b407339d4cac817d07a361f5f3f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:34:03.712715 15936 raft_consensus.cc:1261] T 97ba547488f74ddba2b9cb7e8fc33d52 P 40e66d125d294ad6a81ad404db6c1388 [term 1 FOLLOWER]: Refusing update from remote peer 45c8b407339d4cac817d07a361f5f3f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:34:03.713780 16014 consensus_queue.cc:1035] T 97ba547488f74ddba2b9cb7e8fc33d52 P 45c8b407339d4cac817d07a361f5f3f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a14ff6c94cee4effbbedcd97aae4aec4" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 40179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:34:03.714190 16007 consensus_queue.cc:1035] T 97ba547488f74ddba2b9cb7e8fc33d52 P 45c8b407339d4cac817d07a361f5f3f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40e66d125d294ad6a81ad404db6c1388" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 34631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:34:03.727327 16018 mvcc.cc:205] Tried to move back new op lower bound from 7334764723035525120 to 7334764722731421696. Current Snapshot: MvccSnapshot[applied={T|T < 7334764723035525120}]
I20260929 20:34:03.728451 16017 mvcc.cc:205] Tried to move back new op lower bound from 7334764723035525120 to 7334764722731421696. Current Snapshot: MvccSnapshot[applied={T|T < 7334764723035525120}]
I20260929 20:34:03.729096 16019 mvcc.cc:205] Tried to move back new op lower bound from 7334764723035525120 to 7334764722731421696. Current Snapshot: MvccSnapshot[applied={T|T < 7334764723035525120}]
I20260929 20:34:03.748576 15521 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:34:03.748896 15521 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:34:03.753125 15521 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 277a7093c46843c4bb8fe76c65323260: Sending DeleteTablet for 3 replicas of tablet 97ba547488f74ddba2b9cb7e8fc33d52
I20260929 20:34:03.754503 15781 tablet_service.cc:1514] Processing DeleteTablet for tablet 97ba547488f74ddba2b9cb7e8fc33d52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:34:03 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:57228
I20260929 20:34:03.754504 15646 tablet_service.cc:1514] Processing DeleteTablet for tablet 97ba547488f74ddba2b9cb7e8fc33d52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:34:03 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:59124
I20260929 20:34:03.755242 15916 tablet_service.cc:1514] Processing DeleteTablet for tablet 97ba547488f74ddba2b9cb7e8fc33d52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:34:03 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:49600
I20260929 20:34:03.759131 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 15583
I20260929 20:34:03.765623 16030 tablet_replica.cc:330] T 97ba547488f74ddba2b9cb7e8fc33d52 P 40e66d125d294ad6a81ad404db6c1388: stopping tablet replica
I20260929 20:34:03.766155 16030 raft_consensus.cc:2229] T 97ba547488f74ddba2b9cb7e8fc33d52 P 40e66d125d294ad6a81ad404db6c1388 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:34:03.766510 16028 tablet_replica.cc:330] T 97ba547488f74ddba2b9cb7e8fc33d52 P a14ff6c94cee4effbbedcd97aae4aec4: stopping tablet replica
I20260929 20:34:03.766253 16029 tablet_replica.cc:330] T 97ba547488f74ddba2b9cb7e8fc33d52 P 45c8b407339d4cac817d07a361f5f3f4: stopping tablet replica
I20260929 20:34:03.766741 16030 raft_consensus.cc:2258] T 97ba547488f74ddba2b9cb7e8fc33d52 P 40e66d125d294ad6a81ad404db6c1388 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:34:03.767210 16028 raft_consensus.cc:2229] T 97ba547488f74ddba2b9cb7e8fc33d52 P a14ff6c94cee4effbbedcd97aae4aec4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:34:03.767645 16029 raft_consensus.cc:2229] T 97ba547488f74ddba2b9cb7e8fc33d52 P 45c8b407339d4cac817d07a361f5f3f4 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:34:03.768523 16029 raft_consensus.cc:2258] T 97ba547488f74ddba2b9cb7e8fc33d52 P 45c8b407339d4cac817d07a361f5f3f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:34:03.769064 16028 raft_consensus.cc:2258] T 97ba547488f74ddba2b9cb7e8fc33d52 P a14ff6c94cee4effbbedcd97aae4aec4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:34:03.769588 16030 ts_tablet_manager.cc:1905] T 97ba547488f74ddba2b9cb7e8fc33d52 P 40e66d125d294ad6a81ad404db6c1388: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:34:03.771395 16028 ts_tablet_manager.cc:1905] T 97ba547488f74ddba2b9cb7e8fc33d52 P a14ff6c94cee4effbbedcd97aae4aec4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:34:03.773566 16029 ts_tablet_manager.cc:1905] T 97ba547488f74ddba2b9cb7e8fc33d52 P 45c8b407339d4cac817d07a361f5f3f4: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 20:34:03 dist-test-slave-p29l krb5kdc[15476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.129@KRBTEST.COM
I20260929 20:34:03.777626 16030 ts_tablet_manager.cc:1918] T 97ba547488f74ddba2b9cb7e8fc33d52 P 40e66d125d294ad6a81ad404db6c1388: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:34:03.777808 16030 log.cc:1196] T 97ba547488f74ddba2b9cb7e8fc33d52 P 40e66d125d294ad6a81ad404db6c1388: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-2/wal/wals/97ba547488f74ddba2b9cb7e8fc33d52
I20260929 20:34:03.778400 16030 ts_tablet_manager.cc:1939] T 97ba547488f74ddba2b9cb7e8fc33d52 P 40e66d125d294ad6a81ad404db6c1388: Deleting consensus metadata
I20260929 20:34:03.779364 16028 ts_tablet_manager.cc:1918] T 97ba547488f74ddba2b9cb7e8fc33d52 P a14ff6c94cee4effbbedcd97aae4aec4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:34:03.779560 16028 log.cc:1196] T 97ba547488f74ddba2b9cb7e8fc33d52 P a14ff6c94cee4effbbedcd97aae4aec4: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-1/wal/wals/97ba547488f74ddba2b9cb7e8fc33d52
I20260929 20:34:03.780160 16028 ts_tablet_manager.cc:1939] T 97ba547488f74ddba2b9cb7e8fc33d52 P a14ff6c94cee4effbbedcd97aae4aec4: Deleting consensus metadata
I20260929 20:34:03.780745 15508 catalog_manager.cc:4841] TS 40e66d125d294ad6a81ad404db6c1388 (127.28.244.131:34631): tablet 97ba547488f74ddba2b9cb7e8fc33d52 (table test-table [id=452ad2a0af66434f88376f92e9f4c19c]) successfully deleted
I20260929 20:34:03.782002 15506 catalog_manager.cc:4841] TS a14ff6c94cee4effbbedcd97aae4aec4 (127.28.244.130:40179): tablet 97ba547488f74ddba2b9cb7e8fc33d52 (table test-table [id=452ad2a0af66434f88376f92e9f4c19c]) successfully deleted
I20260929 20:34:03.784611 16029 ts_tablet_manager.cc:1918] T 97ba547488f74ddba2b9cb7e8fc33d52 P 45c8b407339d4cac817d07a361f5f3f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:34:03.784832 16029 log.cc:1196] T 97ba547488f74ddba2b9cb7e8fc33d52 P 45c8b407339d4cac817d07a361f5f3f4: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713721201454-29650-0/minicluster-data/ts-0/wal/wals/97ba547488f74ddba2b9cb7e8fc33d52
I20260929 20:34:03.785425 16029 ts_tablet_manager.cc:1939] T 97ba547488f74ddba2b9cb7e8fc33d52 P 45c8b407339d4cac817d07a361f5f3f4: Deleting consensus metadata
I20260929 20:34:03.785893 15996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:03.760436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59146 (local address 127.28.244.129:38307)
0929 20:34:03.760712 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:03.760722 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:03.763699 (+ 2977us) server_negotiation.cc:407] Connection header received
0929 20:34:03.764824 (+ 1125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:03.764846 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:03.764974 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:03.765070 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:03.767108 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:03.768330 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:03.770329 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:03.770946 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:03.771066 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:03.777079 (+ 6013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:03.777096 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:03.777115 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:03.777174 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:03.781812 (+ 4638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:03.782868 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:03.782879 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:03.782883 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:03.783063 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:03.783570 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:03.783579 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:03.783583 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:03.784203 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:03.784552 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:03.785089 (+ 537us) server_negotiation.cc:299] Negotiation successful
0929 20:34:03.785362 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 20:34:03.786538 15707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:34:03.902261 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 15583
I20260929 20:34:03.902494 15507 catalog_manager.cc:4841] TS 45c8b407339d4cac817d07a361f5f3f4 (127.28.244.129:38307): tablet 97ba547488f74ddba2b9cb7e8fc33d52 (table test-table [id=452ad2a0af66434f88376f92e9f4c19c]) successfully deleted
I20260929 20:34:03.922473 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 15718
Sep 29 20:34:03 dist-test-slave-p29l krb5kdc[15476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.130@KRBTEST.COM
I20260929 20:34:03.938902 15995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:03.923221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57242 (local address 127.28.244.130:40179)
0929 20:34:03.923399 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:03.923406 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:03.923474 (+ 68us) server_negotiation.cc:407] Connection header received
0929 20:34:03.923697 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:03.923707 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:03.923781 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:03.924058 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:03.925361 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:03.926108 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:03.927442 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:03.927920 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:03.928093 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:03.932518 (+ 4425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:03.932534 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:03.932552 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:03.932599 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:03.935528 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:03.936491 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:03.936501 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:03.936504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:03.936657 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:03.937120 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:03.937124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:03.937127 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:03.937559 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:03.937870 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:03.938204 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 20:34:03.938526 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 20:34:03.939600 15842 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:34:04.033177 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 15718
I20260929 20:34:04.052034 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 15853
Sep 29 20:34:04 dist-test-slave-p29l krb5kdc[15476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.131@KRBTEST.COM
I20260929 20:34:04.068397 16013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:04.052880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49616 (local address 127.28.244.131:34631)
0929 20:34:04.053072 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:04.053080 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:04.053109 (+ 29us) server_negotiation.cc:407] Connection header received
0929 20:34:04.053347 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:04.053351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:04.053429 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:04.053539 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:04.054899 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:04.055620 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:04.057162 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:04.057569 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:04.057706 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:04.062672 (+ 4966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:04.062688 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:04.062705 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:04.062745 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:04.065564 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:04.066285 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:04.066293 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:04.066296 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:04.066451 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:04.066798 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:04.066802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:04.066805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:04.067222 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:04.067473 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:04.067854 (+ 381us) server_negotiation.cc:299] Negotiation successful
0929 20:34:04.068052 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:34:04.069085 15976 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:34:04.164377 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 15853
I20260929 20:34:04.182339 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 15489
I20260929 20:34:04.183514 15552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:34:04.295724 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 15489
2026-09-29T20:34:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4594 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T20:34:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:34:04Z Disabled control of system clock
I20260929 20:34:04.378849 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:40427
--webserver_interface=127.28.244.190
--webserver_port=0
--builtin_ntp_servers=127.28.244.148:44461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:40427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 20:34:04.734611 16039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:34:04.734941 16039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:34:04.735110 16039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:34:04.747391 16039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:34:04.747560 16039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:34:04.747633 16039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:34:04.747728 16039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:34:04.747792 16039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:34:04.763108 16039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:44461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:40427
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:40427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:34:04.765107 16039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:34:04.768082 16039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:34:04.782079 16048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:04.782055 16044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:04.782322 16045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:34:04.782923 16039 server_base.cc:1034] running on GCE node
I20260929 20:34:04.783993 16039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:34:04.785755 16039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:34:04.787027 16039 hybrid_clock.cc:648] HybridClock initialized: now 1790714044786985 us; error 90 us; skew 500 ppm
I20260929 20:34:04.787475 16039 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:34:04.790208 16039 webserver.cc:460] Webserver started at http://127.28.244.190:40141/ using document root <none> and password file <none>
I20260929 20:34:04.790997 16039 fs_manager.cc:359] Metadata directory not provided
I20260929 20:34:04.791175 16039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:34:04.791529 16039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:34:04.794993 16039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "95a7b115739d4623adc48fa71137f081"
format_stamp: "Formatted at 2026-09-29 20:34:04 on dist-test-slave-p29l"
server_key: "0a4ce8aee4dd967133dd824e6e2ba304"
server_key_iv: "239794b36823f1af40deeea16423af69"
server_key_version: "encryptionkey@0"
I20260929 20:34:04.795991 16039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "95a7b115739d4623adc48fa71137f081"
format_stamp: "Formatted at 2026-09-29 20:34:04 on dist-test-slave-p29l"
server_key: "0a4ce8aee4dd967133dd824e6e2ba304"
server_key_iv: "239794b36823f1af40deeea16423af69"
server_key_version: "encryptionkey@0"
I20260929 20:34:04.801676 16039 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:34:04.805968 16054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:04.807607 16039 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:34:04.807914 16039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "95a7b115739d4623adc48fa71137f081"
format_stamp: "Formatted at 2026-09-29 20:34:04 on dist-test-slave-p29l"
server_key: "0a4ce8aee4dd967133dd824e6e2ba304"
server_key_iv: "239794b36823f1af40deeea16423af69"
server_key_version: "encryptionkey@0"
I20260929 20:34:04.808192 16039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:34:04.833336 16039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:34:04.834429 16039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:34:04.834757 16039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:34:04.856590 16039 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:40427
I20260929 20:34:04.856601 16105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:40427 every 8 connection(s)
I20260929 20:34:04.858206 16039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:34:04.862581 16106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:34:04.866056 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 16039
I20260929 20:34:04.866276 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:34:04.866775 29650 external_mini_cluster.cc:1428] Setting key 2066c284cef7bc5b19f7a8644401892e
I20260929 20:34:04.875960 16106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95a7b115739d4623adc48fa71137f081: Bootstrap starting.
I20260929 20:34:04.879953 16106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95a7b115739d4623adc48fa71137f081: Neither blocks nor log segments found. Creating new log.
I20260929 20:34:04.881143 16106 log.cc:824] T 00000000000000000000000000000000 P 95a7b115739d4623adc48fa71137f081: Log is configured to *not* fsync() on all Append() calls
I20260929 20:34:04.881635 16109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:04.868735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55438 (local address 127.28.244.190:40427)
0929 20:34:04.869797 (+ 1062us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:04.869808 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:04.869852 (+ 44us) server_negotiation.cc:407] Connection header received
0929 20:34:04.871524 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:04.871547 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:04.872168 (+ 621us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:04.872543 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:04.873295 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:04.874718 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:04.876710 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:04.877457 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:04.877621 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:04.879007 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:04.879015 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:04.879029 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:34:04.879590 (+ 561us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:04.879683 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:34:04.879705 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:04.879800 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:04.880136 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 20:34:04.880529 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":467,"threads_started":1}
I20260929 20:34:04.885039 16106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95a7b115739d4623adc48fa71137f081: No bootstrap required, opened a new log
I20260929 20:34:04.893342 16106 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95a7b115739d4623adc48fa71137f081 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a7b115739d4623adc48fa71137f081" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 40427 } }
I20260929 20:34:04.893934 16106 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95a7b115739d4623adc48fa71137f081 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:34:04.894073 16106 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95a7b115739d4623adc48fa71137f081 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95a7b115739d4623adc48fa71137f081, State: Initialized, Role: FOLLOWER
I20260929 20:34:04.894779 16106 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95a7b115739d4623adc48fa71137f081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a7b115739d4623adc48fa71137f081" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 40427 } }
I20260929 20:34:04.895090 16106 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95a7b115739d4623adc48fa71137f081 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:34:04.895267 16106 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95a7b115739d4623adc48fa71137f081 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:34:04.895462 16106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95a7b115739d4623adc48fa71137f081 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:34:04.897989 16106 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95a7b115739d4623adc48fa71137f081 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a7b115739d4623adc48fa71137f081" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 40427 } }
I20260929 20:34:04.898619 16106 leader_election.cc:302] T 00000000000000000000000000000000 P 95a7b115739d4623adc48fa71137f081 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95a7b115739d4623adc48fa71137f081; no voters:
I20260929 20:34:04.899292 16106 leader_election.cc:288] T 00000000000000000000000000000000 P 95a7b115739d4623adc48fa71137f081 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:34:04.899493 16111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95a7b115739d4623adc48fa71137f081 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:34:04.900354 16111 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95a7b115739d4623adc48fa71137f081 [term 1 LEADER]: Becoming Leader. State: Replica: 95a7b115739d4623adc48fa71137f081, State: Running, Role: LEADER
I20260929 20:34:04.900926 16111 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95a7b115739d4623adc48fa71137f081 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a7b115739d4623adc48fa71137f081" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 40427 } }
I20260929 20:34:04.901741 16106 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95a7b115739d4623adc48fa71137f081 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:34:04.904361 16112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95a7b115739d4623adc48fa71137f081 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95a7b115739d4623adc48fa71137f081" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a7b115739d4623adc48fa71137f081" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 40427 } } }
I20260929 20:34:04.904713 16112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95a7b115739d4623adc48fa71137f081 [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:04.904233 16113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95a7b115739d4623adc48fa71137f081 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95a7b115739d4623adc48fa71137f081. Latest consensus state: current_term: 1 leader_uuid: "95a7b115739d4623adc48fa71137f081" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a7b115739d4623adc48fa71137f081" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 40427 } } }
I20260929 20:34:04.905108 16113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95a7b115739d4623adc48fa71137f081 [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:04.906525 16118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:34:04.912516 16118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:34:04.922814 16118 catalog_manager.cc:1347] Generated new cluster ID: b5852dc9e73543b680d441c777c2fdb8
I20260929 20:34:04.922950 16118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:34:04.942018 16118 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:34:04.943706 16118 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:34:04.950260 16118 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95a7b115739d4623adc48fa71137f081: Generated new TSK 0
I20260929 20:34:04.951435 16118 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:34:04.959347 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.129:0
--local_ip_for_outbound_sockets=127.28.244.129
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:40427
--builtin_ntp_servers=127.28.244.148:44461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 20:34:05.310412 16131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:34:05.310798 16131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:34:05.310966 16131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:34:05.319149 16131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:34:05.319505 16131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:34:05.319723 16131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.129
I20260929 20:34:05.329687 16131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:44461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:40427
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.129
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:34:05.331400 16131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:34:05.333647 16131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:34:05.352747 16136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:05.352653 16137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:05.352751 16139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:34:05.353183 16131 server_base.cc:1034] running on GCE node
I20260929 20:34:05.354225 16131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:34:05.355242 16131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:34:05.356493 16131 hybrid_clock.cc:648] HybridClock initialized: now 1790714045356435 us; error 44 us; skew 500 ppm
I20260929 20:34:05.356926 16131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:34:05.359743 16131 webserver.cc:460] Webserver started at http://127.28.244.129:33927/ using document root <none> and password file <none>
I20260929 20:34:05.360654 16131 fs_manager.cc:359] Metadata directory not provided
I20260929 20:34:05.360790 16131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:34:05.361217 16131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:34:05.364928 16131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-0/data/instance:
uuid: "1da67dcb4752477197b978501341b17f"
format_stamp: "Formatted at 2026-09-29 20:34:05 on dist-test-slave-p29l"
server_key: "b2736fdf950940b69d46ab8edc82a5c5"
server_key_iv: "4274bb00325b3a98b9e134c70edc8c86"
server_key_version: "encryptionkey@0"
I20260929 20:34:05.365913 16131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance:
uuid: "1da67dcb4752477197b978501341b17f"
format_stamp: "Formatted at 2026-09-29 20:34:05 on dist-test-slave-p29l"
server_key: "b2736fdf950940b69d46ab8edc82a5c5"
server_key_iv: "4274bb00325b3a98b9e134c70edc8c86"
server_key_version: "encryptionkey@0"
I20260929 20:34:05.371769 16131 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 20:34:05.376477 16145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:05.378074 16131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260929 20:34:05.378294 16131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-0/wal
uuid: "1da67dcb4752477197b978501341b17f"
format_stamp: "Formatted at 2026-09-29 20:34:05 on dist-test-slave-p29l"
server_key: "b2736fdf950940b69d46ab8edc82a5c5"
server_key_iv: "4274bb00325b3a98b9e134c70edc8c86"
server_key_version: "encryptionkey@0"
I20260929 20:34:05.378547 16131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:34:05.408155 16131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:34:05.409106 16131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:34:05.409438 16131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:34:05.410533 16131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:34:05.412194 16131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:34:05.412330 16131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:05.412478 16131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:34:05.412585 16131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:05.448230 16131 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.129:34327
I20260929 20:34:05.448298 16258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.129:34327 every 8 connection(s)
I20260929 20:34:05.449705 16131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
I20260929 20:34:05.456416 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 16131
I20260929 20:34:05.456632 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance
I20260929 20:34:05.457201 29650 external_mini_cluster.cc:1428] Setting key 985945f5bf236a9cb76c81a4f6a88fef
I20260929 20:34:05.460600 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.130:0
--local_ip_for_outbound_sockets=127.28.244.130
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:40427
--builtin_ntp_servers=127.28.244.148:44461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 20:34:05.464682 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:05.452478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:38263 (local address 127.28.244.190:40427)
0929 20:34:05.452981 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:05.452988 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:05.454318 (+ 1330us) server_negotiation.cc:407] Connection header received
0929 20:34:05.456279 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:05.456284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:05.456366 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:05.456473 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:05.457990 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:05.458828 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:05.461337 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:05.461893 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:05.462015 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:05.463784 (+ 1769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:05.463789 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:05.463792 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:34:05.463874 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:05.463944 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:34:05.463961 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:05.464038 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:05.464316 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 20:34:05.464408 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":245,"threads_started":1}
I20260929 20:34:05.466249 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:05.453161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:40427 (local address 127.28.244.129:38263)
0929 20:34:05.454101 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:05.454146 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:05.455923 (+ 1777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:05.456772 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:05.456785 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:05.456910 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:34:05.457748 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:05.457783 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:05.459023 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:05.459029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:05.461145 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:05.461165 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:05.462162 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:05.462172 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:05.462518 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:05.463377 (+ 859us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:34:05.463382 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:34:05.463448 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:34:05.463565 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:34:05.464068 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:34:05.464081 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:34:05.464092 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 20:34:05.464299 (+ 207us) client_negotiation.cc:240] Negotiation successful
0929 20:34:05.464760 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":286,"threads_started":1}
I20260929 20:34:05.468031 16259 heartbeater.cc:347] Connected to a master server at 127.28.244.190:40427
I20260929 20:34:05.468369 16259 heartbeater.cc:464] Registering TS with master...
I20260929 20:34:05.469203 16259 heartbeater.cc:511] Master 127.28.244.190:40427 requested a full tablet report, sending...
I20260929 20:34:05.471755 16071 ts_manager.cc:194] Registered new tserver with Master: 1da67dcb4752477197b978501341b17f (127.28.244.129:34327)
I20260929 20:34:05.474082 16071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.244.129:38263
W20260929 20:34:05.817189 16263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:34:05.817493 16263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:34:05.817582 16263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:34:05.824998 16263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:34:05.825112 16263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:34:05.825225 16263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.130
I20260929 20:34:05.834859 16263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:44461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:40427
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.130
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:34:05.836360 16263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:34:05.838194 16263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:34:05.852661 16268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:05.852592 16271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:05.852636 16269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:34:05.853368 16263 server_base.cc:1034] running on GCE node
I20260929 20:34:05.854028 16263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:34:05.855062 16263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:34:05.856287 16263 hybrid_clock.cc:648] HybridClock initialized: now 1790714045856198 us; error 82 us; skew 500 ppm
I20260929 20:34:05.856704 16263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:34:05.859597 16263 webserver.cc:460] Webserver started at http://127.28.244.130:36855/ using document root <none> and password file <none>
I20260929 20:34:05.860522 16263 fs_manager.cc:359] Metadata directory not provided
I20260929 20:34:05.860631 16263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:34:05.860984 16263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:34:05.864998 16263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-1/data/instance:
uuid: "cc7cbfc0e10e4355997dde7c08301f4c"
format_stamp: "Formatted at 2026-09-29 20:34:05 on dist-test-slave-p29l"
server_key: "9c77aeecfed3244cc8d4fd6317c80194"
server_key_iv: "15b40c82439c538ae47b6f371e3aca24"
server_key_version: "encryptionkey@0"
I20260929 20:34:05.866042 16263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance:
uuid: "cc7cbfc0e10e4355997dde7c08301f4c"
format_stamp: "Formatted at 2026-09-29 20:34:05 on dist-test-slave-p29l"
server_key: "9c77aeecfed3244cc8d4fd6317c80194"
server_key_iv: "15b40c82439c538ae47b6f371e3aca24"
server_key_version: "encryptionkey@0"
I20260929 20:34:05.872938 16263 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260929 20:34:05.877656 16277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:05.879179 16263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:34:05.879382 16263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-1/wal
uuid: "cc7cbfc0e10e4355997dde7c08301f4c"
format_stamp: "Formatted at 2026-09-29 20:34:05 on dist-test-slave-p29l"
server_key: "9c77aeecfed3244cc8d4fd6317c80194"
server_key_iv: "15b40c82439c538ae47b6f371e3aca24"
server_key_version: "encryptionkey@0"
I20260929 20:34:05.879633 16263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:34:05.899040 16263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:34:05.900058 16263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:34:05.900400 16263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:34:05.901497 16263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:34:05.903184 16263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:34:05.903317 16263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:05.903460 16263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:34:05.903548 16263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:05.938963 16263 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.130:38753
I20260929 20:34:05.939056 16390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.130:38753 every 8 connection(s)
I20260929 20:34:05.940446 16263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
I20260929 20:34:05.942299 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 16263
I20260929 20:34:05.942484 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance
I20260929 20:34:05.942984 29650 external_mini_cluster.cc:1428] Setting key b65d84c6d4f90e66e2fed7493de22bbe
I20260929 20:34:05.946579 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.131:0
--local_ip_for_outbound_sockets=127.28.244.131
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:40427
--builtin_ntp_servers=127.28.244.148:44461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 20:34:05.956975 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:05.943503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:38357 (local address 127.28.244.190:40427)
0929 20:34:05.943734 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:05.943738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:05.945552 (+ 1814us) server_negotiation.cc:407] Connection header received
0929 20:34:05.948488 (+ 2936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:05.948492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:05.948561 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:05.948657 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:05.950205 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:05.951045 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:05.953709 (+ 2664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:05.954127 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:05.954233 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:05.956004 (+ 1771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:05.956009 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:05.956012 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:34:05.956032 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:05.956066 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:34:05.956079 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:05.956200 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:05.956489 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 20:34:05.956602 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 20:34:05.958397 16393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:05.944143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:40427 (local address 127.28.244.130:38357)
0929 20:34:05.945331 (+ 1188us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:05.945426 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:05.948136 (+ 2710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:05.948978 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:05.948992 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:05.949127 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:34:05.950004 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:05.950033 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:05.951212 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:05.951221 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:05.953533 (+ 2312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:05.953551 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:05.954379 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:05.954388 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:05.954734 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:05.955569 (+ 835us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:34:05.955575 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:34:05.955636 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:34:05.955758 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:34:05.956202 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:34:05.956216 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:34:05.956228 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 20:34:05.956454 (+ 226us) client_negotiation.cc:240] Negotiation successful
0929 20:34:05.956906 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":353,"threads_started":1}
I20260929 20:34:05.960074 16391 heartbeater.cc:347] Connected to a master server at 127.28.244.190:40427
I20260929 20:34:05.960475 16391 heartbeater.cc:464] Registering TS with master...
I20260929 20:34:05.961321 16391 heartbeater.cc:511] Master 127.28.244.190:40427 requested a full tablet report, sending...
I20260929 20:34:05.963173 16071 ts_manager.cc:194] Registered new tserver with Master: cc7cbfc0e10e4355997dde7c08301f4c (127.28.244.130:38753)
I20260929 20:34:05.964530 16071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.244.130:38357
W20260929 20:34:06.299410 16394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:34:06.299734 16394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:34:06.299904 16394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:34:06.307682 16394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:34:06.307799 16394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:34:06.307994 16394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.131
I20260929 20:34:06.317627 16394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:44461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:40427
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.131
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:34:06.318956 16394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:34:06.320807 16394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:34:06.334551 16399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:06.334591 16402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:06.334612 16400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:34:06.335351 16394 server_base.cc:1034] running on GCE node
I20260929 20:34:06.336102 16394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:34:06.337134 16394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:34:06.338379 16394 hybrid_clock.cc:648] HybridClock initialized: now 1790714046338298 us; error 71 us; skew 500 ppm
I20260929 20:34:06.338728 16394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:34:06.341444 16394 webserver.cc:460] Webserver started at http://127.28.244.131:41705/ using document root <none> and password file <none>
I20260929 20:34:06.342217 16394 fs_manager.cc:359] Metadata directory not provided
I20260929 20:34:06.342356 16394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:34:06.342684 16394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:34:06.345789 16394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-2/data/instance:
uuid: "5829692ae7ae4ed3bcaf6ab688ef2a98"
format_stamp: "Formatted at 2026-09-29 20:34:06 on dist-test-slave-p29l"
server_key: "6dac7ab80a7a93b48f09291214fba187"
server_key_iv: "a45ac6ffd37de44dc1070537370fb100"
server_key_version: "encryptionkey@0"
I20260929 20:34:06.346669 16394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance:
uuid: "5829692ae7ae4ed3bcaf6ab688ef2a98"
format_stamp: "Formatted at 2026-09-29 20:34:06 on dist-test-slave-p29l"
server_key: "6dac7ab80a7a93b48f09291214fba187"
server_key_iv: "a45ac6ffd37de44dc1070537370fb100"
server_key_version: "encryptionkey@0"
I20260929 20:34:06.352430 16394 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 20:34:06.356442 16408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:06.358048 16394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:34:06.358362 16394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-2/wal
uuid: "5829692ae7ae4ed3bcaf6ab688ef2a98"
format_stamp: "Formatted at 2026-09-29 20:34:06 on dist-test-slave-p29l"
server_key: "6dac7ab80a7a93b48f09291214fba187"
server_key_iv: "a45ac6ffd37de44dc1070537370fb100"
server_key_version: "encryptionkey@0"
I20260929 20:34:06.358635 16394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:34:06.416013 16394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:34:06.416963 16394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:34:06.417305 16394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:34:06.418360 16394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:34:06.419960 16394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:34:06.420089 16394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:06.420233 16394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:34:06.420331 16394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:06.455358 16394 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.131:34665
I20260929 20:34:06.455417 16521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.131:34665 every 8 connection(s)
I20260929 20:34:06.456724 16394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
I20260929 20:34:06.458729 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 16394
I20260929 20:34:06.458927 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance
I20260929 20:34:06.459404 29650 external_mini_cluster.cc:1428] Setting key 478650922050b99ea52303383ed18bad
I20260929 20:34:06.472193 16524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:06.459876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:57701 (local address 127.28.244.190:40427)
0929 20:34:06.460413 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:06.460419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:06.462098 (+ 1679us) server_negotiation.cc:407] Connection header received
0929 20:34:06.464669 (+ 2571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:06.464674 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:06.464790 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:06.464907 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:06.466205 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:06.466996 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:06.468893 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:06.469383 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:06.469508 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:06.471017 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:06.471023 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:06.471026 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:34:06.471049 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:06.471092 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:34:06.471105 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:06.471191 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:06.471480 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 20:34:06.471582 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":276,"threads_started":1}
I20260929 20:34:06.473278 16525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:06.460665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:40427 (local address 127.28.244.131:57701)
0929 20:34:06.461910 (+ 1245us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:06.461967 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:06.464329 (+ 2362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:06.465161 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:06.465171 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:06.465310 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:34:06.465997 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:06.466017 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:06.467164 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:06.467168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:06.468710 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:06.468725 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:06.469623 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:06.469628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:06.469863 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:06.470667 (+ 804us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:34:06.470669 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:34:06.470717 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:34:06.470805 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:34:06.471238 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:34:06.471255 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:34:06.471263 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 20:34:06.471458 (+ 195us) client_negotiation.cc:240] Negotiation successful
0929 20:34:06.471906 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":366,"threads_started":1}
I20260929 20:34:06.474987 16522 heartbeater.cc:347] Connected to a master server at 127.28.244.190:40427
I20260929 20:34:06.475392 16522 heartbeater.cc:464] Registering TS with master...
I20260929 20:34:06.476435 16522 heartbeater.cc:511] Master 127.28.244.190:40427 requested a full tablet report, sending...
I20260929 20:34:06.478099 16071 ts_manager.cc:194] Registered new tserver with Master: 5829692ae7ae4ed3bcaf6ab688ef2a98 (127.28.244.131:34665)
I20260929 20:34:06.478490 16259 heartbeater.cc:503] Master 127.28.244.190:40427 was elected leader, sending a full tablet report...
I20260929 20:34:06.479276 16071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.244.131:57701
I20260929 20:34:06.488664 29650 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:34:06.499485 16524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:06.492908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55444 (local address 127.28.244.190:40427)
0929 20:34:06.493089 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:06.493094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:06.493541 (+ 447us) server_negotiation.cc:407] Connection header received
0929 20:34:06.493667 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:06.493672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:06.493737 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:06.493841 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:06.494456 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:06.495177 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:06.496863 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:06.497291 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:06.497395 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:06.498686 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:06.498690 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:06.498693 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:34:06.498713 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:06.498745 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:34:06.498756 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:06.498879 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:06.499167 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 20:34:06.499218 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 20:34:06.504246 16070 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55444:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:34:06.512270 16070 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:34:06.542361 16536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:06.526194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33228 (local address 127.28.244.130:38753)
0929 20:34:06.530991 (+ 4797us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:06.530997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:06.531023 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:34:06.531103 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:06.531109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:06.531387 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:06.531582 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:34:06.532674 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:06.533759 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:06.537795 (+ 4036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:06.539964 (+ 2169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:06.540154 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:06.540496 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:06.541552 (+ 1056us) server_negotiation.cc:299] Negotiation successful
0929 20:34:06.541818 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4641,"thread_start_us":4520,"threads_started":1}
I20260929 20:34:06.543550 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:06.526098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:38753 (local address 127.0.0.1:33228)
0929 20:34:06.530038 (+ 3940us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:06.530114 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:06.530350 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:06.531916 (+ 1566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:06.531924 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:06.531958 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:34:06.532446 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:06.532462 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:06.533912 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:06.533920 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:06.537494 (+ 3574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:06.537517 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:06.540766 (+ 3249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:06.540776 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:06.541164 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:06.541253 (+ 89us) client_negotiation.cc:769] Sending connection context
0929 20:34:06.542675 (+ 1422us) client_negotiation.cc:240] Negotiation successful
0929 20:34:06.543000 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3540,"mutex_wait_us":33,"thread_start_us":8479,"threads_started":1}
I20260929 20:34:06.547159 16535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:06.526343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:34665 (local address 127.0.0.1:59586)
0929 20:34:06.534973 (+ 8630us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:06.535023 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:06.535164 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:06.536396 (+ 1232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:06.536404 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:06.536436 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:34:06.537047 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:06.537071 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:06.540097 (+ 3026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:06.540103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:06.543182 (+ 3079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:06.543208 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:06.546162 (+ 2954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:06.546169 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:06.546402 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:06.546444 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 20:34:06.546539 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 20:34:06.546711 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8438,"mutex_wait_us":2,"thread_start_us":3103,"threads_started":1}
I20260929 20:34:06.548177 16538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:06.527250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59586 (local address 127.28.244.131:34665)
0929 20:34:06.534444 (+ 7194us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:06.534455 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:06.535321 (+ 866us) server_negotiation.cc:407] Connection header received
0929 20:34:06.535465 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:06.535478 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:06.535879 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:06.536090 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:34:06.537345 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:06.539881 (+ 2536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:06.543528 (+ 3647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:06.545509 (+ 1981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:06.545738 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:06.546005 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:06.547569 (+ 1564us) server_negotiation.cc:299] Negotiation successful
0929 20:34:06.547769 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7009,"thread_start_us":6833,"threads_started":1}
I20260929 20:34:06.549335 16325 tablet_service.cc:1467] Processing CreateTablet for tablet 7f9d5a842d894f4da64b201f451fbab9 (DEFAULT_TABLE table=test-table [id=177914a53432463190db2e129f811e06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:34:06.550873 16325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f9d5a842d894f4da64b201f451fbab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:34:06.551997 16456 tablet_service.cc:1467] Processing CreateTablet for tablet 7f9d5a842d894f4da64b201f451fbab9 (DEFAULT_TABLE table=test-table [id=177914a53432463190db2e129f811e06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:34:06.553732 16456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f9d5a842d894f4da64b201f451fbab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:34:06.554149 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:06.535084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:34327 (local address 127.0.0.1:54412)
0929 20:34:06.543848 (+ 8764us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:06.543874 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:06.544020 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:06.546839 (+ 2819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:06.546845 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:06.546875 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:34:06.547057 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:06.547075 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:06.548981 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:06.548986 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:06.550715 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:06.550732 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:06.553247 (+ 2515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:06.553256 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:06.553468 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:06.553510 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 20:34:06.553622 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 20:34:06.553813 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8620,"thread_start_us":9485,"threads_started":1}
I20260929 20:34:06.554515 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:06.535438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54412 (local address 127.28.244.129:34327)
0929 20:34:06.545862 (+ 10424us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:06.545872 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:06.545907 (+ 35us) server_negotiation.cc:407] Connection header received
0929 20:34:06.546016 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:06.546028 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:06.546356 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:06.546577 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:34:06.547272 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:06.548802 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:06.550925 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:06.552580 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:06.552861 (+ 281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:06.553117 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:06.553694 (+ 577us) server_negotiation.cc:299] Negotiation successful
0929 20:34:06.553964 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10200,"thread_start_us":9775,"threads_started":1}
I20260929 20:34:06.558946 16193 tablet_service.cc:1467] Processing CreateTablet for tablet 7f9d5a842d894f4da64b201f451fbab9 (DEFAULT_TABLE table=test-table [id=177914a53432463190db2e129f811e06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:34:06.560894 16193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f9d5a842d894f4da64b201f451fbab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:34:06.564363 16541 tablet_bootstrap.cc:492] T 7f9d5a842d894f4da64b201f451fbab9 P cc7cbfc0e10e4355997dde7c08301f4c: Bootstrap starting.
I20260929 20:34:06.567504 16541 tablet_bootstrap.cc:654] T 7f9d5a842d894f4da64b201f451fbab9 P cc7cbfc0e10e4355997dde7c08301f4c: Neither blocks nor log segments found. Creating new log.
I20260929 20:34:06.568656 16541 log.cc:824] T 7f9d5a842d894f4da64b201f451fbab9 P cc7cbfc0e10e4355997dde7c08301f4c: Log is configured to *not* fsync() on all Append() calls
I20260929 20:34:06.572394 16543 tablet_bootstrap.cc:492] T 7f9d5a842d894f4da64b201f451fbab9 P 5829692ae7ae4ed3bcaf6ab688ef2a98: Bootstrap starting.
I20260929 20:34:06.573489 16541 tablet_bootstrap.cc:492] T 7f9d5a842d894f4da64b201f451fbab9 P cc7cbfc0e10e4355997dde7c08301f4c: No bootstrap required, opened a new log
I20260929 20:34:06.573796 16541 ts_tablet_manager.cc:1397] T 7f9d5a842d894f4da64b201f451fbab9 P cc7cbfc0e10e4355997dde7c08301f4c: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.001s
I20260929 20:34:06.576537 16543 tablet_bootstrap.cc:654] T 7f9d5a842d894f4da64b201f451fbab9 P 5829692ae7ae4ed3bcaf6ab688ef2a98: Neither blocks nor log segments found. Creating new log.
I20260929 20:34:06.577989 16543 log.cc:824] T 7f9d5a842d894f4da64b201f451fbab9 P 5829692ae7ae4ed3bcaf6ab688ef2a98: Log is configured to *not* fsync() on all Append() calls
I20260929 20:34:06.578812 16544 tablet_bootstrap.cc:492] T 7f9d5a842d894f4da64b201f451fbab9 P 1da67dcb4752477197b978501341b17f: Bootstrap starting.
I20260929 20:34:06.583148 16543 tablet_bootstrap.cc:492] T 7f9d5a842d894f4da64b201f451fbab9 P 5829692ae7ae4ed3bcaf6ab688ef2a98: No bootstrap required, opened a new log
I20260929 20:34:06.582808 16541 raft_consensus.cc:348] T 7f9d5a842d894f4da64b201f451fbab9 P cc7cbfc0e10e4355997dde7c08301f4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc7cbfc0e10e4355997dde7c08301f4c" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 38753 } } peers { permanent_uuid: "5829692ae7ae4ed3bcaf6ab688ef2a98" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 34665 } } peers { permanent_uuid: "1da67dcb4752477197b978501341b17f" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34327 } }
I20260929 20:34:06.583369 16541 raft_consensus.cc:374] T 7f9d5a842d894f4da64b201f451fbab9 P cc7cbfc0e10e4355997dde7c08301f4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:34:06.583395 16544 tablet_bootstrap.cc:654] T 7f9d5a842d894f4da64b201f451fbab9 P 1da67dcb4752477197b978501341b17f: Neither blocks nor log segments found. Creating new log.
I20260929 20:34:06.583500 16543 ts_tablet_manager.cc:1397] T 7f9d5a842d894f4da64b201f451fbab9 P 5829692ae7ae4ed3bcaf6ab688ef2a98: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260929 20:34:06.583508 16541 raft_consensus.cc:729] T 7f9d5a842d894f4da64b201f451fbab9 P cc7cbfc0e10e4355997dde7c08301f4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc7cbfc0e10e4355997dde7c08301f4c, State: Initialized, Role: FOLLOWER
I20260929 20:34:06.584298 16541 consensus_queue.cc:260] T 7f9d5a842d894f4da64b201f451fbab9 P cc7cbfc0e10e4355997dde7c08301f4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc7cbfc0e10e4355997dde7c08301f4c" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 38753 } } peers { permanent_uuid: "5829692ae7ae4ed3bcaf6ab688ef2a98" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 34665 } } peers { permanent_uuid: "1da67dcb4752477197b978501341b17f" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34327 } }
I20260929 20:34:06.585017 16544 log.cc:824] T 7f9d5a842d894f4da64b201f451fbab9 P 1da67dcb4752477197b978501341b17f: Log is configured to *not* fsync() on all Append() calls
I20260929 20:34:06.585713 16391 heartbeater.cc:503] Master 127.28.244.190:40427 was elected leader, sending a full tablet report...
I20260929 20:34:06.586413 16541 ts_tablet_manager.cc:1428] T 7f9d5a842d894f4da64b201f451fbab9 P cc7cbfc0e10e4355997dde7c08301f4c: Time spent starting tablet: real 0.012s user 0.010s sys 0.004s
I20260929 20:34:06.589651 16544 tablet_bootstrap.cc:492] T 7f9d5a842d894f4da64b201f451fbab9 P 1da67dcb4752477197b978501341b17f: No bootstrap required, opened a new log
I20260929 20:34:06.589982 16544 ts_tablet_manager.cc:1397] T 7f9d5a842d894f4da64b201f451fbab9 P 1da67dcb4752477197b978501341b17f: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260929 20:34:06.594530 16543 raft_consensus.cc:348] T 7f9d5a842d894f4da64b201f451fbab9 P 5829692ae7ae4ed3bcaf6ab688ef2a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc7cbfc0e10e4355997dde7c08301f4c" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 38753 } } peers { permanent_uuid: "5829692ae7ae4ed3bcaf6ab688ef2a98" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 34665 } } peers { permanent_uuid: "1da67dcb4752477197b978501341b17f" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34327 } }
I20260929 20:34:06.595105 16543 raft_consensus.cc:374] T 7f9d5a842d894f4da64b201f451fbab9 P 5829692ae7ae4ed3bcaf6ab688ef2a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:34:06.595227 16543 raft_consensus.cc:729] T 7f9d5a842d894f4da64b201f451fbab9 P 5829692ae7ae4ed3bcaf6ab688ef2a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5829692ae7ae4ed3bcaf6ab688ef2a98, State: Initialized, Role: FOLLOWER
I20260929 20:34:06.595959 16543 consensus_queue.cc:260] T 7f9d5a842d894f4da64b201f451fbab9 P 5829692ae7ae4ed3bcaf6ab688ef2a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc7cbfc0e10e4355997dde7c08301f4c" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 38753 } } peers { permanent_uuid: "5829692ae7ae4ed3bcaf6ab688ef2a98" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 34665 } } peers { permanent_uuid: "1da67dcb4752477197b978501341b17f" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34327 } }
I20260929 20:34:06.597201 16522 heartbeater.cc:503] Master 127.28.244.190:40427 was elected leader, sending a full tablet report...
I20260929 20:34:06.597990 16543 ts_tablet_manager.cc:1428] T 7f9d5a842d894f4da64b201f451fbab9 P 5829692ae7ae4ed3bcaf6ab688ef2a98: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260929 20:34:06.599272 16544 raft_consensus.cc:348] T 7f9d5a842d894f4da64b201f451fbab9 P 1da67dcb4752477197b978501341b17f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc7cbfc0e10e4355997dde7c08301f4c" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 38753 } } peers { permanent_uuid: "5829692ae7ae4ed3bcaf6ab688ef2a98" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 34665 } } peers { permanent_uuid: "1da67dcb4752477197b978501341b17f" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34327 } }
I20260929 20:34:06.599779 16544 raft_consensus.cc:374] T 7f9d5a842d894f4da64b201f451fbab9 P 1da67dcb4752477197b978501341b17f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:34:06.599948 16544 raft_consensus.cc:729] T 7f9d5a842d894f4da64b201f451fbab9 P 1da67dcb4752477197b978501341b17f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1da67dcb4752477197b978501341b17f, State: Initialized, Role: FOLLOWER
I20260929 20:34:06.600616 16544 consensus_queue.cc:260] T 7f9d5a842d894f4da64b201f451fbab9 P 1da67dcb4752477197b978501341b17f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc7cbfc0e10e4355997dde7c08301f4c" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 38753 } } peers { permanent_uuid: "5829692ae7ae4ed3bcaf6ab688ef2a98" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 34665 } } peers { permanent_uuid: "1da67dcb4752477197b978501341b17f" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34327 } }
I20260929 20:34:06.602412 16544 ts_tablet_manager.cc:1428] T 7f9d5a842d894f4da64b201f451fbab9 P 1da67dcb4752477197b978501341b17f: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260929 20:34:06.611984 16549 raft_consensus.cc:482] T 7f9d5a842d894f4da64b201f451fbab9 P 1da67dcb4752477197b978501341b17f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:34:06.612270 16549 raft_consensus.cc:504] T 7f9d5a842d894f4da64b201f451fbab9 P 1da67dcb4752477197b978501341b17f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc7cbfc0e10e4355997dde7c08301f4c" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 38753 } } peers { permanent_uuid: "5829692ae7ae4ed3bcaf6ab688ef2a98" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 34665 } } peers { permanent_uuid: "1da67dcb4752477197b978501341b17f" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34327 } }
I20260929 20:34:06.613968 16549 leader_election.cc:288] T 7f9d5a842d894f4da64b201f451fbab9 P 1da67dcb4752477197b978501341b17f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc7cbfc0e10e4355997dde7c08301f4c (127.28.244.130:38753), 5829692ae7ae4ed3bcaf6ab688ef2a98 (127.28.244.131:34665)
I20260929 20:34:06.623510 16536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:06.614488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:57041 (local address 127.28.244.130:38753)
0929 20:34:06.614703 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:06.614711 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:06.615784 (+ 1073us) server_negotiation.cc:407] Connection header received
0929 20:34:06.616012 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:06.616018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:06.616106 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:06.616187 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:34:06.616829 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:06.617857 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:06.620962 (+ 3105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:06.622227 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:06.622420 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:06.622496 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:06.623093 (+ 597us) server_negotiation.cc:299] Negotiation successful
0929 20:34:06.623241 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:34:06.623536 16538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:06.614512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:39721 (local address 127.28.244.131:34665)
0929 20:34:06.614711 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:06.614717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:06.615450 (+ 733us) server_negotiation.cc:407] Connection header received
0929 20:34:06.615675 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:06.615680 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:06.615853 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:06.615969 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:34:06.616602 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:06.617735 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:06.620966 (+ 3231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:06.622250 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:06.622420 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:06.622496 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:06.623093 (+ 597us) server_negotiation.cc:299] Negotiation successful
0929 20:34:06.623241 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 20:34:06.623798 16550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:06.614441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:34665 (local address 127.28.244.129:39721)
0929 20:34:06.615297 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:06.615328 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:06.615518 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:06.616004 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:06.616007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:06.616028 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:34:06.616417 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:06.616429 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:06.617905 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:06.617909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:06.620678 (+ 2769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:06.620698 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:06.622521 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:06.622526 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:06.622852 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:06.622917 (+ 65us) client_negotiation.cc:769] Sending connection context
0929 20:34:06.623006 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 20:34:06.623248 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"mutex_wait_us":302,"thread_start_us":444,"threads_started":1}
I20260929 20:34:06.623957 16551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:06.614784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:38753 (local address 127.28.244.129:57041)
0929 20:34:06.615688 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:06.615701 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:06.615792 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:06.616223 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:06.616229 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:06.616260 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:34:06.616671 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:06.616680 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:06.617990 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:06.617996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:06.620678 (+ 2682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:06.620698 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:06.622506 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:06.622515 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:06.622856 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:06.622893 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 20:34:06.623006 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 20:34:06.623248 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"mutex_wait_us":295,"thread_start_us":527,"threads_started":1}
I20260929 20:34:06.624917 16476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f9d5a842d894f4da64b201f451fbab9" candidate_uuid: "1da67dcb4752477197b978501341b17f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5829692ae7ae4ed3bcaf6ab688ef2a98" is_pre_election: true
I20260929 20:34:06.625012 16346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f9d5a842d894f4da64b201f451fbab9" candidate_uuid: "1da67dcb4752477197b978501341b17f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7cbfc0e10e4355997dde7c08301f4c" is_pre_election: true
I20260929 20:34:06.625478 16476 raft_consensus.cc:2454] T 7f9d5a842d894f4da64b201f451fbab9 P 5829692ae7ae4ed3bcaf6ab688ef2a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1da67dcb4752477197b978501341b17f in term 0.
I20260929 20:34:06.625486 16346 raft_consensus.cc:2454] T 7f9d5a842d894f4da64b201f451fbab9 P cc7cbfc0e10e4355997dde7c08301f4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1da67dcb4752477197b978501341b17f in term 0.
I20260929 20:34:06.626206 16147 leader_election.cc:302] T 7f9d5a842d894f4da64b201f451fbab9 P 1da67dcb4752477197b978501341b17f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1da67dcb4752477197b978501341b17f, 5829692ae7ae4ed3bcaf6ab688ef2a98, cc7cbfc0e10e4355997dde7c08301f4c; no voters:
I20260929 20:34:06.626622 16549 raft_consensus.cc:2781] T 7f9d5a842d894f4da64b201f451fbab9 P 1da67dcb4752477197b978501341b17f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:34:06.626768 16549 raft_consensus.cc:482] T 7f9d5a842d894f4da64b201f451fbab9 P 1da67dcb4752477197b978501341b17f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:34:06.626895 16549 raft_consensus.cc:3037] T 7f9d5a842d894f4da64b201f451fbab9 P 1da67dcb4752477197b978501341b17f [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:34:06.629247 16549 raft_consensus.cc:504] T 7f9d5a842d894f4da64b201f451fbab9 P 1da67dcb4752477197b978501341b17f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc7cbfc0e10e4355997dde7c08301f4c" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 38753 } } peers { permanent_uuid: "5829692ae7ae4ed3bcaf6ab688ef2a98" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 34665 } } peers { permanent_uuid: "1da67dcb4752477197b978501341b17f" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34327 } }
I20260929 20:34:06.630060 16549 leader_election.cc:288] T 7f9d5a842d894f4da64b201f451fbab9 P 1da67dcb4752477197b978501341b17f [CANDIDATE]: Term 1 election: Requested vote from peers cc7cbfc0e10e4355997dde7c08301f4c (127.28.244.130:38753), 5829692ae7ae4ed3bcaf6ab688ef2a98 (127.28.244.131:34665)
I20260929 20:34:06.630589 16346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f9d5a842d894f4da64b201f451fbab9" candidate_uuid: "1da67dcb4752477197b978501341b17f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7cbfc0e10e4355997dde7c08301f4c"
I20260929 20:34:06.630692 16476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f9d5a842d894f4da64b201f451fbab9" candidate_uuid: "1da67dcb4752477197b978501341b17f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5829692ae7ae4ed3bcaf6ab688ef2a98"
I20260929 20:34:06.630844 16346 raft_consensus.cc:3037] T 7f9d5a842d894f4da64b201f451fbab9 P cc7cbfc0e10e4355997dde7c08301f4c [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:34:06.631034 16476 raft_consensus.cc:3037] T 7f9d5a842d894f4da64b201f451fbab9 P 5829692ae7ae4ed3bcaf6ab688ef2a98 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:34:06.633360 16346 raft_consensus.cc:2454] T 7f9d5a842d894f4da64b201f451fbab9 P cc7cbfc0e10e4355997dde7c08301f4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1da67dcb4752477197b978501341b17f in term 1.
I20260929 20:34:06.633435 16476 raft_consensus.cc:2454] T 7f9d5a842d894f4da64b201f451fbab9 P 5829692ae7ae4ed3bcaf6ab688ef2a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1da67dcb4752477197b978501341b17f in term 1.
I20260929 20:34:06.633909 16147 leader_election.cc:302] T 7f9d5a842d894f4da64b201f451fbab9 P 1da67dcb4752477197b978501341b17f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1da67dcb4752477197b978501341b17f, cc7cbfc0e10e4355997dde7c08301f4c; no voters:
I20260929 20:34:06.634306 16549 raft_consensus.cc:2781] T 7f9d5a842d894f4da64b201f451fbab9 P 1da67dcb4752477197b978501341b17f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:34:06.634789 16549 raft_consensus.cc:686] T 7f9d5a842d894f4da64b201f451fbab9 P 1da67dcb4752477197b978501341b17f [term 1 LEADER]: Becoming Leader. State: Replica: 1da67dcb4752477197b978501341b17f, State: Running, Role: LEADER
I20260929 20:34:06.635353 16549 consensus_queue.cc:237] T 7f9d5a842d894f4da64b201f451fbab9 P 1da67dcb4752477197b978501341b17f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc7cbfc0e10e4355997dde7c08301f4c" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 38753 } } peers { permanent_uuid: "5829692ae7ae4ed3bcaf6ab688ef2a98" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 34665 } } peers { permanent_uuid: "1da67dcb4752477197b978501341b17f" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34327 } }
I20260929 20:34:06.641197 16070 catalog_manager.cc:5495] T 7f9d5a842d894f4da64b201f451fbab9 P 1da67dcb4752477197b978501341b17f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1da67dcb4752477197b978501341b17f (127.28.244.129). New cstate: current_term: 1 leader_uuid: "1da67dcb4752477197b978501341b17f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc7cbfc0e10e4355997dde7c08301f4c" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 38753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5829692ae7ae4ed3bcaf6ab688ef2a98" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 34665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1da67dcb4752477197b978501341b17f" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 34327 } health_report { overall_health: HEALTHY } } }
I20260929 20:34:06.669457 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:06.663151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54424 (local address 127.28.244.129:34327)
0929 20:34:06.663323 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:06.663328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:06.663452 (+ 124us) server_negotiation.cc:407] Connection header received
0929 20:34:06.663634 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:06.663638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:06.663778 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:06.664005 (+ 227us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:34:06.664373 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:06.665162 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:06.666686 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:06.667544 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:06.667723 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:06.668356 (+ 633us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:34:06.668916 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:34:06.669062 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:06.669193 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 20:34:06.669235 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 20:34:06.678766 16346 raft_consensus.cc:1261] T 7f9d5a842d894f4da64b201f451fbab9 P cc7cbfc0e10e4355997dde7c08301f4c [term 1 FOLLOWER]: Refusing update from remote peer 1da67dcb4752477197b978501341b17f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:34:06.678768 16476 raft_consensus.cc:1261] T 7f9d5a842d894f4da64b201f451fbab9 P 5829692ae7ae4ed3bcaf6ab688ef2a98 [term 1 FOLLOWER]: Refusing update from remote peer 1da67dcb4752477197b978501341b17f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:34:06.679925 16549 consensus_queue.cc:1035] T 7f9d5a842d894f4da64b201f451fbab9 P 1da67dcb4752477197b978501341b17f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5829692ae7ae4ed3bcaf6ab688ef2a98" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 34665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:34:06.680259 16552 consensus_queue.cc:1035] T 7f9d5a842d894f4da64b201f451fbab9 P 1da67dcb4752477197b978501341b17f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7cbfc0e10e4355997dde7c08301f4c" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 38753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 20:34:06.693310 16392 tablet.cc:2369] T 7f9d5a842d894f4da64b201f451fbab9 P cc7cbfc0e10e4355997dde7c08301f4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:34:06.693919 16555 mvcc.cc:205] Tried to move back new op lower bound from 7334764735185649664 to 7334764735023255552. Current Snapshot: MvccSnapshot[applied={T|T < 7334764735185649664}]
I20260929 20:34:06.694136 16557 mvcc.cc:205] Tried to move back new op lower bound from 7334764735185649664 to 7334764735023255552. Current Snapshot: MvccSnapshot[applied={T|T < 7334764735185649664}]
I20260929 20:34:06.696722 16556 mvcc.cc:205] Tried to move back new op lower bound from 7334764735185649664 to 7334764735023255552. Current Snapshot: MvccSnapshot[applied={T|T < 7334764735185649664}]
I20260929 20:34:06.709664 16070 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:34:06.709954 16070 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:34:06.714298 16070 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 95a7b115739d4623adc48fa71137f081: Sending DeleteTablet for 3 replicas of tablet 7f9d5a842d894f4da64b201f451fbab9
I20260929 20:34:06.715792 16325 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f9d5a842d894f4da64b201f451fbab9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:34:06 UTC) from {username='slave'} at 127.0.0.1:33228
I20260929 20:34:06.716076 16193 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f9d5a842d894f4da64b201f451fbab9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:34:06 UTC) from {username='slave'} at 127.0.0.1:54412
I20260929 20:34:06.716634 16456 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f9d5a842d894f4da64b201f451fbab9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:34:06 UTC) from {username='slave'} at 127.0.0.1:59586
I20260929 20:34:06.716959 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 16131
I20260929 20:34:06.727103 16564 tablet_replica.cc:330] T 7f9d5a842d894f4da64b201f451fbab9 P cc7cbfc0e10e4355997dde7c08301f4c: stopping tablet replica
I20260929 20:34:06.728757 16564 raft_consensus.cc:2229] T 7f9d5a842d894f4da64b201f451fbab9 P cc7cbfc0e10e4355997dde7c08301f4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:34:06.730563 16564 raft_consensus.cc:2258] T 7f9d5a842d894f4da64b201f451fbab9 P cc7cbfc0e10e4355997dde7c08301f4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:34:06.731045 16565 tablet_replica.cc:330] T 7f9d5a842d894f4da64b201f451fbab9 P 1da67dcb4752477197b978501341b17f: stopping tablet replica
I20260929 20:34:06.731576 16565 raft_consensus.cc:2229] T 7f9d5a842d894f4da64b201f451fbab9 P 1da67dcb4752477197b978501341b17f [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:34:06.732781 16565 raft_consensus.cc:2258] T 7f9d5a842d894f4da64b201f451fbab9 P 1da67dcb4752477197b978501341b17f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:34:06.734961 16564 ts_tablet_manager.cc:1905] T 7f9d5a842d894f4da64b201f451fbab9 P cc7cbfc0e10e4355997dde7c08301f4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:34:06.735764 16566 tablet_replica.cc:330] T 7f9d5a842d894f4da64b201f451fbab9 P 5829692ae7ae4ed3bcaf6ab688ef2a98: stopping tablet replica
I20260929 20:34:06.736267 16566 raft_consensus.cc:2229] T 7f9d5a842d894f4da64b201f451fbab9 P 5829692ae7ae4ed3bcaf6ab688ef2a98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:34:06.736869 16566 raft_consensus.cc:2258] T 7f9d5a842d894f4da64b201f451fbab9 P 5829692ae7ae4ed3bcaf6ab688ef2a98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:34:06.737697 16565 ts_tablet_manager.cc:1905] T 7f9d5a842d894f4da64b201f451fbab9 P 1da67dcb4752477197b978501341b17f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:34:06.742259 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:06.718490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54438 (local address 127.28.244.129:34327)
0929 20:34:06.718741 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:06.718750 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:06.730482 (+ 11732us) server_negotiation.cc:407] Connection header received
0929 20:34:06.731054 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:06.731070 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:06.731191 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:06.731278 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:06.732079 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:06.733316 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:06.737315 (+ 3999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:06.738044 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:06.738164 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:06.740087 (+ 1923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:06.740101 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:06.740125 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:34:06.741150 (+ 1025us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:06.741222 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:34:06.741255 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:06.741371 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:06.741684 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 20:34:06.741839 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 20:34:06.744038 16566 ts_tablet_manager.cc:1905] T 7f9d5a842d894f4da64b201f451fbab9 P 5829692ae7ae4ed3bcaf6ab688ef2a98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:34:06.744530 16254 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:34:06.749586 16564 ts_tablet_manager.cc:1918] T 7f9d5a842d894f4da64b201f451fbab9 P cc7cbfc0e10e4355997dde7c08301f4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:34:06.749790 16564 log.cc:1196] T 7f9d5a842d894f4da64b201f451fbab9 P cc7cbfc0e10e4355997dde7c08301f4c: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-1/wal/wals/7f9d5a842d894f4da64b201f451fbab9
I20260929 20:34:06.750375 16564 ts_tablet_manager.cc:1939] T 7f9d5a842d894f4da64b201f451fbab9 P cc7cbfc0e10e4355997dde7c08301f4c: Deleting consensus metadata
I20260929 20:34:06.752261 16566 ts_tablet_manager.cc:1918] T 7f9d5a842d894f4da64b201f451fbab9 P 5829692ae7ae4ed3bcaf6ab688ef2a98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:34:06.752599 16566 log.cc:1196] T 7f9d5a842d894f4da64b201f451fbab9 P 5829692ae7ae4ed3bcaf6ab688ef2a98: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-2/wal/wals/7f9d5a842d894f4da64b201f451fbab9
I20260929 20:34:06.752643 16056 catalog_manager.cc:4841] TS cc7cbfc0e10e4355997dde7c08301f4c (127.28.244.130:38753): tablet 7f9d5a842d894f4da64b201f451fbab9 (table test-table [id=177914a53432463190db2e129f811e06]) successfully deleted
I20260929 20:34:06.753188 16566 ts_tablet_manager.cc:1939] T 7f9d5a842d894f4da64b201f451fbab9 P 5829692ae7ae4ed3bcaf6ab688ef2a98: Deleting consensus metadata
I20260929 20:34:06.754930 16057 catalog_manager.cc:4841] TS 5829692ae7ae4ed3bcaf6ab688ef2a98 (127.28.244.131:34665): tablet 7f9d5a842d894f4da64b201f451fbab9 (table test-table [id=177914a53432463190db2e129f811e06]) successfully deleted
I20260929 20:34:06.847599 16565 ts_tablet_manager.cc:1918] T 7f9d5a842d894f4da64b201f451fbab9 P 1da67dcb4752477197b978501341b17f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:34:06.847896 16565 log.cc:1196] T 7f9d5a842d894f4da64b201f451fbab9 P 1da67dcb4752477197b978501341b17f: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713721201454-29650-0/minicluster-data/ts-0/wal/wals/7f9d5a842d894f4da64b201f451fbab9
I20260929 20:34:06.848446 16565 ts_tablet_manager.cc:1939] T 7f9d5a842d894f4da64b201f451fbab9 P 1da67dcb4752477197b978501341b17f: Deleting consensus metadata
I20260929 20:34:06.850315 16056 catalog_manager.cc:4841] TS 1da67dcb4752477197b978501341b17f (127.28.244.129:34327): tablet 7f9d5a842d894f4da64b201f451fbab9 (table test-table [id=177914a53432463190db2e129f811e06]) successfully deleted
I20260929 20:34:06.850685 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 16131
I20260929 20:34:06.875700 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 16263
I20260929 20:34:06.884068 16536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:06.876556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33232 (local address 127.28.244.130:38753)
0929 20:34:06.876743 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:06.876749 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:06.876770 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:34:06.876933 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:06.876939 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:06.877072 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:06.877188 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:06.877722 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:06.878874 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:06.880338 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:06.881020 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:06.881165 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:06.882297 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:06.882309 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:06.882329 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:34:06.883102 (+ 773us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:06.883192 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:34:06.883224 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:06.883345 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:06.883482 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 20:34:06.883604 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 20:34:06.884899 16386 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:34:06.975052 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 16263
I20260929 20:34:06.993002 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 16394
I20260929 20:34:06.999938 16538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:06.993825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59600 (local address 127.28.244.131:34665)
0929 20:34:06.994060 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:06.994065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:06.994084 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:34:06.994218 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:06.994223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:06.994322 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:06.994415 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:06.994809 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:06.995589 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:06.996930 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:06.997317 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:06.997412 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:06.998478 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:06.998487 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:06.998503 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:34:06.999121 (+ 618us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:06.999171 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:34:06.999196 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:06.999283 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:06.999458 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 20:34:06.999555 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260929 20:34:07.000644 16517 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:34:07.093199 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 16394
I20260929 20:34:07.111037 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 16039
I20260929 20:34:07.112262 16101 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:34:07.224222 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 16039
2026-09-29T20:34:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2927 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:34:07 dist-test-slave-p29l krb5kdc[16574](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:34:07 dist-test-slave-p29l krb5kdc[16574](info): set up 2 sockets
Sep 29 20:34:07 dist-test-slave-p29l krb5kdc[16574](info): commencing operation
krb5kdc: starting...
W20260929 20:34:08.379794 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.006s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:34:08 dist-test-slave-p29l krb5kdc[16574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:34:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:34:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.244.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
I20260929 20:34:08.655428 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:34001
--webserver_interface=127.28.244.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:36311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:34001
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:34:09.013078 16587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:34:09.013480 16587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:34:09.013659 16587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:34:09.021346 16587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:34:09.021543 16587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:34:09.021660 16587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:34:09.021772 16587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:34:09.021860 16587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:34:09.032407 16587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:36311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:34001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:34001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:34:09.033942 16587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:34:09.035341 16587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:34:09.048074 16593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:09.048153 16595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:09.048329 16592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:34:09.049500 16587 server_base.cc:1034] running on GCE node
I20260929 20:34:09.050429 16587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:34:09.052093 16587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:34:09.053378 16587 hybrid_clock.cc:648] HybridClock initialized: now 1790714049053260 us; error 59 us; skew 500 ppm
I20260929 20:34:09.053818 16587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:34:09 dist-test-slave-p29l krb5kdc[16574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714049, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:34:09.057469 16587 init.cc:377] Logged in from keytab as kudu/127.28.244.190@KRBTEST.COM (short username kudu)
I20260929 20:34:09.059695 16587 webserver.cc:460] Webserver started at http://127.28.244.190:39433/ using document root <none> and password file <none>
I20260929 20:34:09.060560 16587 fs_manager.cc:359] Metadata directory not provided
I20260929 20:34:09.060734 16587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:34:09.061079 16587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:34:09.064666 16587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "d7373441ce3141e3a200b243eeca2723"
format_stamp: "Formatted at 2026-09-29 20:34:09 on dist-test-slave-p29l"
server_key: "1b06a4098c4430e365b6f99a45e9a9cd"
server_key_iv: "3239afa8284b4280bf300c33af9c0d77"
server_key_version: "encryptionkey@0"
I20260929 20:34:09.065641 16587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "d7373441ce3141e3a200b243eeca2723"
format_stamp: "Formatted at 2026-09-29 20:34:09 on dist-test-slave-p29l"
server_key: "1b06a4098c4430e365b6f99a45e9a9cd"
server_key_iv: "3239afa8284b4280bf300c33af9c0d77"
server_key_version: "encryptionkey@0"
I20260929 20:34:09.071791 16587 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 20:34:09.076203 16602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:09.077870 16587 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 20:34:09.078064 16587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "d7373441ce3141e3a200b243eeca2723"
format_stamp: "Formatted at 2026-09-29 20:34:09 on dist-test-slave-p29l"
server_key: "1b06a4098c4430e365b6f99a45e9a9cd"
server_key_iv: "3239afa8284b4280bf300c33af9c0d77"
server_key_version: "encryptionkey@0"
I20260929 20:34:09.078272 16587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:34:09.099172 16587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:34:09.104576 16587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:34:09.104857 16587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:34:09.126003 16587 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:34001
I20260929 20:34:09.126025 16654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:34001 every 8 connection(s)
I20260929 20:34:09.127529 16587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:34:09.132099 16655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:34:09.133417 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 16587
I20260929 20:34:09.133627 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:34:09.134153 29650 external_mini_cluster.cc:1428] Setting key 312c8e23a66e1ac94f9cd3b06fc383e7
I20260929 20:34:09.149499 16655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7373441ce3141e3a200b243eeca2723: Bootstrap starting.
Sep 29 20:34:09 dist-test-slave-p29l krb5kdc[16574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:34:09.154889 16655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7373441ce3141e3a200b243eeca2723: Neither blocks nor log segments found. Creating new log.
I20260929 20:34:09.156855 16655 log.cc:824] T 00000000000000000000000000000000 P d7373441ce3141e3a200b243eeca2723: Log is configured to *not* fsync() on all Append() calls
I20260929 20:34:09.161993 16655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7373441ce3141e3a200b243eeca2723: No bootstrap required, opened a new log
I20260929 20:34:09.167016 16658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:09.136410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36440 (local address 127.28.244.190:34001)
0929 20:34:09.137344 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:09.137361 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:09.137422 (+ 61us) server_negotiation.cc:407] Connection header received
0929 20:34:09.139511 (+ 2089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:09.139545 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:09.140190 (+ 645us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:09.140706 (+ 516us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:09.142232 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:09.144405 (+ 2173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:09.146530 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:09.147976 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:09.148234 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:09.155108 (+ 6874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:09.155123 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:09.155151 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:09.155204 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:09.159949 (+ 4745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:09.161381 (+ 1432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:09.161392 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:09.161404 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:09.161634 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:09.162231 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:09.162237 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:09.162240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:09.163065 (+ 825us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:09.164680 (+ 1615us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:09.165051 (+ 371us) server_negotiation.cc:299] Negotiation successful
0929 20:34:09.165449 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":363,"threads_started":1}
I20260929 20:34:09.173846 16655 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7373441ce3141e3a200b243eeca2723 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7373441ce3141e3a200b243eeca2723" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34001 } }
I20260929 20:34:09.174316 16655 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7373441ce3141e3a200b243eeca2723 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:34:09.174449 16655 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7373441ce3141e3a200b243eeca2723 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7373441ce3141e3a200b243eeca2723, State: Initialized, Role: FOLLOWER
I20260929 20:34:09.175272 16655 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7373441ce3141e3a200b243eeca2723 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7373441ce3141e3a200b243eeca2723" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34001 } }
I20260929 20:34:09.175572 16655 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7373441ce3141e3a200b243eeca2723 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:34:09.175715 16655 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7373441ce3141e3a200b243eeca2723 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:34:09.175987 16655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7373441ce3141e3a200b243eeca2723 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:34:09.179061 16655 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7373441ce3141e3a200b243eeca2723 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7373441ce3141e3a200b243eeca2723" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34001 } }
I20260929 20:34:09.179661 16655 leader_election.cc:302] T 00000000000000000000000000000000 P d7373441ce3141e3a200b243eeca2723 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7373441ce3141e3a200b243eeca2723; no voters:
I20260929 20:34:09.180351 16655 leader_election.cc:288] T 00000000000000000000000000000000 P d7373441ce3141e3a200b243eeca2723 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:34:09.180522 16660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7373441ce3141e3a200b243eeca2723 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:34:09.181793 16660 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7373441ce3141e3a200b243eeca2723 [term 1 LEADER]: Becoming Leader. State: Replica: d7373441ce3141e3a200b243eeca2723, State: Running, Role: LEADER
I20260929 20:34:09.182653 16660 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7373441ce3141e3a200b243eeca2723 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7373441ce3141e3a200b243eeca2723" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34001 } }
I20260929 20:34:09.182865 16655 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7373441ce3141e3a200b243eeca2723 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:34:09.186082 16661 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7373441ce3141e3a200b243eeca2723 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7373441ce3141e3a200b243eeca2723" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7373441ce3141e3a200b243eeca2723" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34001 } } }
I20260929 20:34:09.186554 16661 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7373441ce3141e3a200b243eeca2723 [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:09.187038 16662 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7373441ce3141e3a200b243eeca2723 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7373441ce3141e3a200b243eeca2723. Latest consensus state: current_term: 1 leader_uuid: "d7373441ce3141e3a200b243eeca2723" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7373441ce3141e3a200b243eeca2723" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34001 } } }
I20260929 20:34:09.187551 16662 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7373441ce3141e3a200b243eeca2723 [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:09.188071 16669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:34:09.193717 16669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:34:09.203478 16669 catalog_manager.cc:1347] Generated new cluster ID: 90315ee006fc454c98d2d6839dd926fb
I20260929 20:34:09.203629 16669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:34:09.214502 16669 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:34:09.216634 16669 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:34:09.227998 16669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7373441ce3141e3a200b243eeca2723: Generated new TSK 0
I20260929 20:34:09.229282 16669 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.244.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab.
Entry for principal kudu/127.28.244.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab.
I20260929 20:34:09.296927 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.129:0
--local_ip_for_outbound_sockets=127.28.244.129
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:34001
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:36311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:34:09.660111 16681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:34:09.660427 16681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:34:09.660583 16681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:34:09.669822 16681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:34:09.669956 16681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:34:09.670152 16681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.129
I20260929 20:34:09.680771 16681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:36311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.129.keytab
--principal=kudu/127.28.244.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:34001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.129
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:34:09.682323 16681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:34:09.683983 16681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:34:09.698488 16687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:09.698439 16686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:09.698452 16689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:34:09.699332 16681 server_base.cc:1034] running on GCE node
I20260929 20:34:09.700119 16681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:34:09.701224 16681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:34:09.702504 16681 hybrid_clock.cc:648] HybridClock initialized: now 1790714049702430 us; error 75 us; skew 500 ppm
I20260929 20:34:09.702889 16681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:34:09 dist-test-slave-p29l krb5kdc[16574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714049, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:34:09.706486 16681 init.cc:377] Logged in from keytab as kudu/127.28.244.129@KRBTEST.COM (short username kudu)
I20260929 20:34:09.708156 16681 webserver.cc:460] Webserver started at http://127.28.244.129:43607/ using document root <none> and password file <none>
I20260929 20:34:09.708861 16681 fs_manager.cc:359] Metadata directory not provided
I20260929 20:34:09.708979 16681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:34:09.709259 16681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:34:09.712379 16681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-0/data/instance:
uuid: "23f4e90639fe4e4eb1e00cafb603c22b"
format_stamp: "Formatted at 2026-09-29 20:34:09 on dist-test-slave-p29l"
server_key: "b36f8a43da0a3c4f18293fb649cc39ec"
server_key_iv: "b4a6805ec7bbe4f4332b04cc663ef70f"
server_key_version: "encryptionkey@0"
I20260929 20:34:09.713227 16681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance:
uuid: "23f4e90639fe4e4eb1e00cafb603c22b"
format_stamp: "Formatted at 2026-09-29 20:34:09 on dist-test-slave-p29l"
server_key: "b36f8a43da0a3c4f18293fb649cc39ec"
server_key_iv: "b4a6805ec7bbe4f4332b04cc663ef70f"
server_key_version: "encryptionkey@0"
I20260929 20:34:09.718890 16681 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:34:09.722710 16696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:09.724344 16681 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:34:09.724550 16681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-0/wal
uuid: "23f4e90639fe4e4eb1e00cafb603c22b"
format_stamp: "Formatted at 2026-09-29 20:34:09 on dist-test-slave-p29l"
server_key: "b36f8a43da0a3c4f18293fb649cc39ec"
server_key_iv: "b4a6805ec7bbe4f4332b04cc663ef70f"
server_key_version: "encryptionkey@0"
I20260929 20:34:09.724773 16681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:34:09.747092 16681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:34:09.751629 16681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:34:09.752035 16681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:34:09.753160 16681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:34:09.754591 16681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:34:09.754698 16681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:09.754858 16681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:34:09.754938 16681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:09.790102 16681 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.129:40499
I20260929 20:34:09.790153 16808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.129:40499 every 8 connection(s)
I20260929 20:34:09.791488 16681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
I20260929 20:34:09.794909 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 16681
I20260929 20:34:09.795140 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance
I20260929 20:34:09.795919 29650 external_mini_cluster.cc:1428] Setting key 9945a069f02016653203159c63e613c6
Sep 29 20:34:09 dist-test-slave-p29l krb5kdc[16574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714049, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.129@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:34:09.813407 16812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:09.794217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:46295 (local address 127.28.244.190:34001)
0929 20:34:09.796479 (+ 2262us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:09.796486 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:09.796511 (+ 25us) server_negotiation.cc:407] Connection header received
0929 20:34:09.798163 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:09.798168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:09.798274 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:09.798396 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:09.800117 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:09.800873 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:09.802541 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:09.802969 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:09.803119 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:09.806979 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:09.806986 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:09.806989 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:09.807026 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:09.809739 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:09.810665 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:09.810671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:09.810674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:09.810869 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:09.811394 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:09.811398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:09.811400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:09.811696 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:09.811906 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:09.812729 (+ 823us) server_negotiation.cc:299] Negotiation successful
0929 20:34:09.813023 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2130,"thread_start_us":2002,"threads_started":1}
I20260929 20:34:09.814518 16813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:09.794729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:34001 (local address 127.28.244.129:46295)
0929 20:34:09.795542 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:09.795591 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:09.797875 (+ 2284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:09.798659 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:09.798668 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:09.799216 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:34:09.799933 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:09.799955 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:09.801057 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:09.801062 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:09.802372 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:09.802384 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:09.803232 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:09.803241 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:09.803522 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:09.804341 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:34:09.804373 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:34:09.806693 (+ 2320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:34:09.809928 (+ 3235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:09.809938 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:09.809960 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:09.810494 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:09.811003 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:09.811008 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:09.811012 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:09.811241 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:09.811878 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:34:09.811888 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:34:09.812305 (+ 417us) client_negotiation.cc:769] Sending connection context
0929 20:34:09.812599 (+ 294us) client_negotiation.cc:240] Negotiation successful
0929 20:34:09.813221 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":325,"threads_started":1}
I20260929 20:34:09.816411 16809 heartbeater.cc:347] Connected to a master server at 127.28.244.190:34001
I20260929 20:34:09.816749 16809 heartbeater.cc:464] Registering TS with master...
I20260929 20:34:09.817524 16809 heartbeater.cc:511] Master 127.28.244.190:34001 requested a full tablet report, sending...
I20260929 20:34:09.820127 16619 ts_manager.cc:194] Registered new tserver with Master: 23f4e90639fe4e4eb1e00cafb603c22b (127.28.244.129:40499)
WARNING: no policy specified for kudu/127.28.244.130@KRBTEST.COM; defaulting to no policy
I20260929 20:34:09.821933 16619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.129@KRBTEST.COM'} at 127.28.244.129:46295
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab.
Entry for principal kudu/127.28.244.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab.
I20260929 20:34:09.856335 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.130:0
--local_ip_for_outbound_sockets=127.28.244.130
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:34001
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:36311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:34:10.211577 16816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:34:10.211911 16816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:34:10.212009 16816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:34:10.219362 16816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:34:10.219496 16816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:34:10.219614 16816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.130
I20260929 20:34:10.230021 16816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:36311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.130.keytab
--principal=kudu/127.28.244.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:34001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.130
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:34:10.231417 16816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:34:10.233036 16816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:34:10.246912 16821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:10.246999 16824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:10.247105 16822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:34:10.248148 16816 server_base.cc:1034] running on GCE node
I20260929 20:34:10.248802 16816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:34:10.249871 16816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:34:10.251132 16816 hybrid_clock.cc:648] HybridClock initialized: now 1790714050251073 us; error 51 us; skew 500 ppm
I20260929 20:34:10.251492 16816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:34:10 dist-test-slave-p29l krb5kdc[16574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714050, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:34:10.254890 16816 init.cc:377] Logged in from keytab as kudu/127.28.244.130@KRBTEST.COM (short username kudu)
I20260929 20:34:10.256503 16816 webserver.cc:460] Webserver started at http://127.28.244.130:44349/ using document root <none> and password file <none>
I20260929 20:34:10.257295 16816 fs_manager.cc:359] Metadata directory not provided
I20260929 20:34:10.257473 16816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:34:10.257889 16816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:34:10.261376 16816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-1/data/instance:
uuid: "7184d42327ca488fb6d3ae6f561ec3ee"
format_stamp: "Formatted at 2026-09-29 20:34:10 on dist-test-slave-p29l"
server_key: "f049404d0b72cf04e663cba20ba773fa"
server_key_iv: "c6d9988c834a3d7c94b11e98286b38a6"
server_key_version: "encryptionkey@0"
I20260929 20:34:10.262226 16816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance:
uuid: "7184d42327ca488fb6d3ae6f561ec3ee"
format_stamp: "Formatted at 2026-09-29 20:34:10 on dist-test-slave-p29l"
server_key: "f049404d0b72cf04e663cba20ba773fa"
server_key_iv: "c6d9988c834a3d7c94b11e98286b38a6"
server_key_version: "encryptionkey@0"
I20260929 20:34:10.268023 16816 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 20:34:10.271926 16831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:10.273460 16816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:34:10.273655 16816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-1/wal
uuid: "7184d42327ca488fb6d3ae6f561ec3ee"
format_stamp: "Formatted at 2026-09-29 20:34:10 on dist-test-slave-p29l"
server_key: "f049404d0b72cf04e663cba20ba773fa"
server_key_iv: "c6d9988c834a3d7c94b11e98286b38a6"
server_key_version: "encryptionkey@0"
I20260929 20:34:10.273873 16816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:34:10.288520 16816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:34:10.293428 16816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:34:10.293689 16816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:34:10.294678 16816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:34:10.296118 16816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:34:10.296197 16816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:10.296284 16816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:34:10.296331 16816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:10.330057 16816 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.130:44807
I20260929 20:34:10.330106 16945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.130:44807 every 8 connection(s)
I20260929 20:34:10.331437 16816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
I20260929 20:34:10.333622 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 16816
I20260929 20:34:10.333824 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance
I20260929 20:34:10.334389 29650 external_mini_cluster.cc:1428] Setting key da636a672158e52ecc49e188218d59d0
Sep 29 20:34:10 dist-test-slave-p29l krb5kdc[16574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714050, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.130@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:34:10.359344 16948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:10.334306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:34117 (local address 127.28.244.190:34001)
0929 20:34:10.334907 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:10.334911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:10.336011 (+ 1100us) server_negotiation.cc:407] Connection header received
0929 20:34:10.338954 (+ 2943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:10.338959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:10.339031 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:10.339130 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:10.341428 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:10.342171 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:10.344821 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:10.345277 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:10.345382 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:10.349889 (+ 4507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:10.349900 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:10.349908 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:10.349960 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:10.354075 (+ 4115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:10.355824 (+ 1749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:10.355834 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:10.355841 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:10.356051 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:10.356714 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:10.356722 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:10.356729 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:10.357165 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:10.357332 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:10.358524 (+ 1192us) server_negotiation.cc:299] Negotiation successful
0929 20:34:10.358823 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":352,"threads_started":1}
I20260929 20:34:10.360044 16949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:10.334971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:34001 (local address 127.28.244.130:34117)
0929 20:34:10.335843 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:10.335899 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:10.338570 (+ 2671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:10.339446 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:10.339462 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:10.340325 (+ 863us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:34:10.341217 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:10.341247 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:10.342333 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:10.342342 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:10.344660 (+ 2318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:10.344680 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:10.345524 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:10.345533 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:10.345874 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:10.346800 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:34:10.346852 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:34:10.349513 (+ 2661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:34:10.354343 (+ 4830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:10.354356 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:10.354419 (+ 63us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:10.355576 (+ 1157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:10.356222 (+ 646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:10.356231 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:10.356237 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:10.356551 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:10.357308 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:34:10.357321 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:34:10.357888 (+ 567us) client_negotiation.cc:769] Sending connection context
0929 20:34:10.358273 (+ 385us) client_negotiation.cc:240] Negotiation successful
0929 20:34:10.358775 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":359,"threads_started":1}
I20260929 20:34:10.361673 16946 heartbeater.cc:347] Connected to a master server at 127.28.244.190:34001
I20260929 20:34:10.362062 16946 heartbeater.cc:464] Registering TS with master...
I20260929 20:34:10.363096 16946 heartbeater.cc:511] Master 127.28.244.190:34001 requested a full tablet report, sending...
I20260929 20:34:10.364996 16619 ts_manager.cc:194] Registered new tserver with Master: 7184d42327ca488fb6d3ae6f561ec3ee (127.28.244.130:44807)
I20260929 20:34:10.365875 16619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.130@KRBTEST.COM'} at 127.28.244.130:34117
WARNING: no policy specified for kudu/127.28.244.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab.
Entry for principal kudu/127.28.244.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab.
I20260929 20:34:10.400074 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.131:0
--local_ip_for_outbound_sockets=127.28.244.131
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:34001
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:36311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:34:10.764338 16952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:34:10.764698 16952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:34:10.764868 16952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:34:10.772668 16952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:34:10.772842 16952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:34:10.773016 16952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.131
I20260929 20:34:10.783628 16952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:36311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.131.keytab
--principal=kudu/127.28.244.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:34001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.131
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:34:10.785105 16952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:34:10.786511 16952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:34:10.800537 16960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:10.800630 16957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:10.800603 16958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:34:10.801556 16952 server_base.cc:1034] running on GCE node
I20260929 20:34:10.802333 16952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:34:10.803400 16952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:34:10.804675 16952 hybrid_clock.cc:648] HybridClock initialized: now 1790714050804622 us; error 47 us; skew 500 ppm
I20260929 20:34:10.805073 16952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:34:10 dist-test-slave-p29l krb5kdc[16574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714050, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:34:10.808697 16952 init.cc:377] Logged in from keytab as kudu/127.28.244.131@KRBTEST.COM (short username kudu)
I20260929 20:34:10.810400 16952 webserver.cc:460] Webserver started at http://127.28.244.131:41719/ using document root <none> and password file <none>
I20260929 20:34:10.811178 16952 fs_manager.cc:359] Metadata directory not provided
I20260929 20:34:10.811337 16952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:34:10.811667 16952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:34:10.814819 16952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-2/data/instance:
uuid: "0448da810b044f5981a7da545f948f1f"
format_stamp: "Formatted at 2026-09-29 20:34:10 on dist-test-slave-p29l"
server_key: "9845e52a99bd047f13517c2f36c85050"
server_key_iv: "14affac5e453e3e61347039f86e8d01c"
server_key_version: "encryptionkey@0"
I20260929 20:34:10.815675 16952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance:
uuid: "0448da810b044f5981a7da545f948f1f"
format_stamp: "Formatted at 2026-09-29 20:34:10 on dist-test-slave-p29l"
server_key: "9845e52a99bd047f13517c2f36c85050"
server_key_iv: "14affac5e453e3e61347039f86e8d01c"
server_key_version: "encryptionkey@0"
I20260929 20:34:10.821435 16952 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 20:34:10.825380 16967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:10.825508 16809 heartbeater.cc:503] Master 127.28.244.190:34001 was elected leader, sending a full tablet report...
I20260929 20:34:10.827112 16952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:34:10.827342 16952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-2/wal
uuid: "0448da810b044f5981a7da545f948f1f"
format_stamp: "Formatted at 2026-09-29 20:34:10 on dist-test-slave-p29l"
server_key: "9845e52a99bd047f13517c2f36c85050"
server_key_iv: "14affac5e453e3e61347039f86e8d01c"
server_key_version: "encryptionkey@0"
I20260929 20:34:10.827627 16952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:34:10.844696 16952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:34:10.849483 16952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:34:10.849787 16952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:34:10.850826 16952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:34:10.852320 16952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:34:10.852415 16952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:10.852605 16952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:34:10.852672 16952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:10.888269 16952 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.131:42265
I20260929 20:34:10.888317 17080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.131:42265 every 8 connection(s)
I20260929 20:34:10.889679 16952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
I20260929 20:34:10.897178 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 16952
I20260929 20:34:10.897383 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance
I20260929 20:34:10.897823 29650 external_mini_cluster.cc:1428] Setting key b26fcf00b3972e55397b56051ce27a7a
Sep 29 20:34:10 dist-test-slave-p29l krb5kdc[16574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714050, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.131@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:34:10.911852 17083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:10.892772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:51857 (local address 127.28.244.190:34001)
0929 20:34:10.893295 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:10.893300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:10.894566 (+ 1266us) server_negotiation.cc:407] Connection header received
0929 20:34:10.896597 (+ 2031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:10.896602 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:10.896693 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:10.896855 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:10.898632 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:10.899584 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:10.901241 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:10.901662 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:10.901797 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:10.905654 (+ 3857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:10.905661 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:10.905664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:10.905701 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:10.908403 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:10.909284 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:10.909289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:10.909291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:10.909430 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:10.909912 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:10.909916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:10.909919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:10.910201 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:10.910376 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:10.911134 (+ 758us) server_negotiation.cc:299] Negotiation successful
0929 20:34:10.911382 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":255,"threads_started":1}
I20260929 20:34:10.913106 17084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:10.893476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:34001 (local address 127.28.244.131:51857)
0929 20:34:10.894329 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:10.894394 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:10.896287 (+ 1893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:10.897070 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:10.897079 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:10.897750 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:34:10.898428 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:10.898448 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:10.899766 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:10.899770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:10.901085 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:10.901099 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:10.901905 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:10.901913 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:10.902161 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:10.902900 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:34:10.902925 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:34:10.905393 (+ 2468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:34:10.908586 (+ 3193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:10.908597 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:10.908617 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:10.909126 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:10.909554 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:10.909560 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:10.909564 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:10.909775 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:10.910338 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:34:10.910348 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:34:10.910739 (+ 391us) client_negotiation.cc:769] Sending connection context
0929 20:34:10.911027 (+ 288us) client_negotiation.cc:240] Negotiation successful
0929 20:34:10.911611 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":328,"threads_started":1}
I20260929 20:34:10.914565 17081 heartbeater.cc:347] Connected to a master server at 127.28.244.190:34001
I20260929 20:34:10.914922 17081 heartbeater.cc:464] Registering TS with master...
I20260929 20:34:10.915719 17081 heartbeater.cc:511] Master 127.28.244.190:34001 requested a full tablet report, sending...
I20260929 20:34:10.917237 16619 ts_manager.cc:194] Registered new tserver with Master: 0448da810b044f5981a7da545f948f1f (127.28.244.131:42265)
I20260929 20:34:10.918104 16619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.244.131@KRBTEST.COM'} at 127.28.244.131:51857
I20260929 20:34:10.926735 29650 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:34:10.945727 17083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:10.931195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36450 (local address 127.28.244.190:34001)
0929 20:34:10.931469 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:10.931475 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:10.931797 (+ 322us) server_negotiation.cc:407] Connection header received
0929 20:34:10.932018 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:10.932022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:10.932094 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:10.932186 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:10.933945 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:10.934694 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:10.936243 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:10.936686 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:10.936831 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:10.939494 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:10.939500 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:10.939503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:10.939545 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:10.942685 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:10.943577 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:10.943582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:10.943585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:10.943719 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:10.944204 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:10.944209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:10.944211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:10.944585 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:10.944783 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:10.945096 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 20:34:10.945322 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260929 20:34:10.950155 16619 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:34:10.957384 16619 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:34:10.994769 17098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:10.970718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:42265 (local address 127.0.0.1:55230)
0929 20:34:10.981568 (+ 10850us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:10.981654 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:10.981949 (+ 295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:10.985436 (+ 3487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:10.985449 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:10.985496 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:34:10.986043 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:10.986061 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:10.988622 (+ 2561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:10.988627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:10.991301 (+ 2674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:10.991316 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:10.993665 (+ 2349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:10.993672 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:10.993891 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:10.993949 (+ 58us) client_negotiation.cc:769] Sending connection context
0929 20:34:10.994100 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 20:34:10.994314 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10384,"mutex_wait_us":1,"thread_start_us":14457,"threads_started":1}
I20260929 20:34:10.995661 17096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:10.970896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55230 (local address 127.28.244.131:42265)
0929 20:34:10.984106 (+ 13210us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:10.984120 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:10.984160 (+ 40us) server_negotiation.cc:407] Connection header received
0929 20:34:10.984319 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:10.984331 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:10.984698 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:10.984980 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:34:10.986588 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:10.988426 (+ 1838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:10.991491 (+ 3065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:10.993086 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:10.993304 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:10.993518 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:10.995058 (+ 1540us) server_negotiation.cc:299] Negotiation successful
0929 20:34:10.995260 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12960,"thread_start_us":15343,"threads_started":1}
I20260929 20:34:10.995980 17094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:10.973655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:40499 (local address 127.0.0.1:52688)
0929 20:34:10.984908 (+ 11253us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:10.984933 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:10.985095 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:10.988344 (+ 3249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:10.988353 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:10.988381 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:34:10.989029 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:10.989046 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:10.990812 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:10.990824 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:10.992688 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:10.992704 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:10.995021 (+ 2317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:10.995026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:10.995188 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:10.995219 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 20:34:10.995325 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 20:34:10.995488 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11067,"mutex_wait_us":34,"thread_start_us":7253,"threads_started":1}
I20260929 20:34:10.996485 17099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:10.974439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52688 (local address 127.28.244.129:40499)
0929 20:34:10.987753 (+ 13314us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:10.987761 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:10.987787 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:34:10.987928 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:10.987935 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:10.988149 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:10.988448 (+ 299us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:34:10.989244 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:10.990676 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:10.992921 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:10.994451 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:10.994667 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:10.994860 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:10.995580 (+ 720us) server_negotiation.cc:299] Negotiation successful
0929 20:34:10.995986 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13154,"thread_start_us":13003,"threads_started":1}
I20260929 20:34:10.996585 17097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:10.971114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33740 (local address 127.28.244.130:44807)
0929 20:34:10.987332 (+ 16218us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:10.987343 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:10.987382 (+ 39us) server_negotiation.cc:407] Connection header received
0929 20:34:10.987494 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:10.987502 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:10.987797 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:10.988059 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:34:10.988931 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:10.990552 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:10.993324 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:10.994809 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:10.994982 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:10.995143 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:10.996086 (+ 943us) server_negotiation.cc:299] Negotiation successful
0929 20:34:10.996263 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16063,"thread_start_us":15943,"threads_started":1}
I20260929 20:34:10.997052 17095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:10.970868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:44807 (local address 127.0.0.1:33740)
0929 20:34:10.983162 (+ 12294us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:10.983185 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:10.983370 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:10.988092 (+ 4722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:10.988097 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:10.988127 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:34:10.988653 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:10.988669 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:10.990706 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:10.990713 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:10.993126 (+ 2413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:10.993147 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:10.995301 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:10.995308 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:10.995518 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:10.995552 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 20:34:10.995686 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 20:34:10.995870 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12145,"mutex_wait_us":106,"thread_start_us":11975,"threads_started":1}
I20260929 20:34:11.000108 16744 tablet_service.cc:1467] Processing CreateTablet for tablet d2d9c58fa06b40299379a6787cf60d33 (DEFAULT_TABLE table=test-table [id=f4e7e6872e5f47d7a0a0d847f343c99a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:34:11.000105 17016 tablet_service.cc:1467] Processing CreateTablet for tablet d2d9c58fa06b40299379a6787cf60d33 (DEFAULT_TABLE table=test-table [id=f4e7e6872e5f47d7a0a0d847f343c99a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:34:11.000501 16880 tablet_service.cc:1467] Processing CreateTablet for tablet d2d9c58fa06b40299379a6787cf60d33 (DEFAULT_TABLE table=test-table [id=f4e7e6872e5f47d7a0a0d847f343c99a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:34:11.001703 16880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2d9c58fa06b40299379a6787cf60d33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:34:11.001720 16744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2d9c58fa06b40299379a6787cf60d33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:34:11.001720 17016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2d9c58fa06b40299379a6787cf60d33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:34:11.014196 17100 tablet_bootstrap.cc:492] T d2d9c58fa06b40299379a6787cf60d33 P 7184d42327ca488fb6d3ae6f561ec3ee: Bootstrap starting.
I20260929 20:34:11.016860 17101 tablet_bootstrap.cc:492] T d2d9c58fa06b40299379a6787cf60d33 P 23f4e90639fe4e4eb1e00cafb603c22b: Bootstrap starting.
I20260929 20:34:11.019729 17100 tablet_bootstrap.cc:654] T d2d9c58fa06b40299379a6787cf60d33 P 7184d42327ca488fb6d3ae6f561ec3ee: Neither blocks nor log segments found. Creating new log.
I20260929 20:34:11.020624 17101 tablet_bootstrap.cc:654] T d2d9c58fa06b40299379a6787cf60d33 P 23f4e90639fe4e4eb1e00cafb603c22b: Neither blocks nor log segments found. Creating new log.
I20260929 20:34:11.021163 17100 log.cc:824] T d2d9c58fa06b40299379a6787cf60d33 P 7184d42327ca488fb6d3ae6f561ec3ee: Log is configured to *not* fsync() on all Append() calls
I20260929 20:34:11.022053 17101 log.cc:824] T d2d9c58fa06b40299379a6787cf60d33 P 23f4e90639fe4e4eb1e00cafb603c22b: Log is configured to *not* fsync() on all Append() calls
I20260929 20:34:11.022854 17102 tablet_bootstrap.cc:492] T d2d9c58fa06b40299379a6787cf60d33 P 0448da810b044f5981a7da545f948f1f: Bootstrap starting.
I20260929 20:34:11.025141 17100 tablet_bootstrap.cc:492] T d2d9c58fa06b40299379a6787cf60d33 P 7184d42327ca488fb6d3ae6f561ec3ee: No bootstrap required, opened a new log
I20260929 20:34:11.025489 17100 ts_tablet_manager.cc:1397] T d2d9c58fa06b40299379a6787cf60d33 P 7184d42327ca488fb6d3ae6f561ec3ee: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20260929 20:34:11.027602 17102 tablet_bootstrap.cc:654] T d2d9c58fa06b40299379a6787cf60d33 P 0448da810b044f5981a7da545f948f1f: Neither blocks nor log segments found. Creating new log.
I20260929 20:34:11.028251 17101 tablet_bootstrap.cc:492] T d2d9c58fa06b40299379a6787cf60d33 P 23f4e90639fe4e4eb1e00cafb603c22b: No bootstrap required, opened a new log
I20260929 20:34:11.028548 17101 ts_tablet_manager.cc:1397] T d2d9c58fa06b40299379a6787cf60d33 P 23f4e90639fe4e4eb1e00cafb603c22b: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.009s
I20260929 20:34:11.029037 17102 log.cc:824] T d2d9c58fa06b40299379a6787cf60d33 P 0448da810b044f5981a7da545f948f1f: Log is configured to *not* fsync() on all Append() calls
I20260929 20:34:11.031982 17102 tablet_bootstrap.cc:492] T d2d9c58fa06b40299379a6787cf60d33 P 0448da810b044f5981a7da545f948f1f: No bootstrap required, opened a new log
I20260929 20:34:11.032296 17102 ts_tablet_manager.cc:1397] T d2d9c58fa06b40299379a6787cf60d33 P 0448da810b044f5981a7da545f948f1f: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.002s
I20260929 20:34:11.036693 17100 raft_consensus.cc:348] T d2d9c58fa06b40299379a6787cf60d33 P 7184d42327ca488fb6d3ae6f561ec3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0448da810b044f5981a7da545f948f1f" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 42265 } } peers { permanent_uuid: "7184d42327ca488fb6d3ae6f561ec3ee" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 44807 } } peers { permanent_uuid: "23f4e90639fe4e4eb1e00cafb603c22b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 40499 } }
I20260929 20:34:11.037238 17100 raft_consensus.cc:374] T d2d9c58fa06b40299379a6787cf60d33 P 7184d42327ca488fb6d3ae6f561ec3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:34:11.037343 17100 raft_consensus.cc:729] T d2d9c58fa06b40299379a6787cf60d33 P 7184d42327ca488fb6d3ae6f561ec3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7184d42327ca488fb6d3ae6f561ec3ee, State: Initialized, Role: FOLLOWER
I20260929 20:34:11.038174 17100 consensus_queue.cc:260] T d2d9c58fa06b40299379a6787cf60d33 P 7184d42327ca488fb6d3ae6f561ec3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0448da810b044f5981a7da545f948f1f" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 42265 } } peers { permanent_uuid: "7184d42327ca488fb6d3ae6f561ec3ee" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 44807 } } peers { permanent_uuid: "23f4e90639fe4e4eb1e00cafb603c22b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 40499 } }
I20260929 20:34:11.039564 16946 heartbeater.cc:503] Master 127.28.244.190:34001 was elected leader, sending a full tablet report...
I20260929 20:34:11.039724 17101 raft_consensus.cc:348] T d2d9c58fa06b40299379a6787cf60d33 P 23f4e90639fe4e4eb1e00cafb603c22b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0448da810b044f5981a7da545f948f1f" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 42265 } } peers { permanent_uuid: "7184d42327ca488fb6d3ae6f561ec3ee" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 44807 } } peers { permanent_uuid: "23f4e90639fe4e4eb1e00cafb603c22b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 40499 } }
I20260929 20:34:11.040272 17101 raft_consensus.cc:374] T d2d9c58fa06b40299379a6787cf60d33 P 23f4e90639fe4e4eb1e00cafb603c22b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:34:11.040414 17100 ts_tablet_manager.cc:1428] T d2d9c58fa06b40299379a6787cf60d33 P 7184d42327ca488fb6d3ae6f561ec3ee: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260929 20:34:11.040402 17101 raft_consensus.cc:729] T d2d9c58fa06b40299379a6787cf60d33 P 23f4e90639fe4e4eb1e00cafb603c22b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23f4e90639fe4e4eb1e00cafb603c22b, State: Initialized, Role: FOLLOWER
I20260929 20:34:11.041081 17101 consensus_queue.cc:260] T d2d9c58fa06b40299379a6787cf60d33 P 23f4e90639fe4e4eb1e00cafb603c22b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0448da810b044f5981a7da545f948f1f" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 42265 } } peers { permanent_uuid: "7184d42327ca488fb6d3ae6f561ec3ee" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 44807 } } peers { permanent_uuid: "23f4e90639fe4e4eb1e00cafb603c22b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 40499 } }
I20260929 20:34:11.042086 17102 raft_consensus.cc:348] T d2d9c58fa06b40299379a6787cf60d33 P 0448da810b044f5981a7da545f948f1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0448da810b044f5981a7da545f948f1f" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 42265 } } peers { permanent_uuid: "7184d42327ca488fb6d3ae6f561ec3ee" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 44807 } } peers { permanent_uuid: "23f4e90639fe4e4eb1e00cafb603c22b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 40499 } }
I20260929 20:34:11.042586 17102 raft_consensus.cc:374] T d2d9c58fa06b40299379a6787cf60d33 P 0448da810b044f5981a7da545f948f1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:34:11.042691 17102 raft_consensus.cc:729] T d2d9c58fa06b40299379a6787cf60d33 P 0448da810b044f5981a7da545f948f1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0448da810b044f5981a7da545f948f1f, State: Initialized, Role: FOLLOWER
I20260929 20:34:11.043419 17102 consensus_queue.cc:260] T d2d9c58fa06b40299379a6787cf60d33 P 0448da810b044f5981a7da545f948f1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0448da810b044f5981a7da545f948f1f" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 42265 } } peers { permanent_uuid: "7184d42327ca488fb6d3ae6f561ec3ee" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 44807 } } peers { permanent_uuid: "23f4e90639fe4e4eb1e00cafb603c22b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 40499 } }
I20260929 20:34:11.043852 17101 ts_tablet_manager.cc:1428] T d2d9c58fa06b40299379a6787cf60d33 P 23f4e90639fe4e4eb1e00cafb603c22b: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260929 20:34:11.044386 16810 tablet.cc:2369] T d2d9c58fa06b40299379a6787cf60d33 P 23f4e90639fe4e4eb1e00cafb603c22b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:34:11.045928 17081 heartbeater.cc:503] Master 127.28.244.190:34001 was elected leader, sending a full tablet report...
I20260929 20:34:11.047056 17102 ts_tablet_manager.cc:1428] T d2d9c58fa06b40299379a6787cf60d33 P 0448da810b044f5981a7da545f948f1f: Time spent starting tablet: real 0.014s user 0.009s sys 0.003s
W20260929 20:34:11.083657 16947 tablet.cc:2369] T d2d9c58fa06b40299379a6787cf60d33 P 7184d42327ca488fb6d3ae6f561ec3ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:34:11.141556 17082 tablet.cc:2369] T d2d9c58fa06b40299379a6787cf60d33 P 0448da810b044f5981a7da545f948f1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:34:11.163132 17106 raft_consensus.cc:482] T d2d9c58fa06b40299379a6787cf60d33 P 7184d42327ca488fb6d3ae6f561ec3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:34:11.163529 17106 raft_consensus.cc:504] T d2d9c58fa06b40299379a6787cf60d33 P 7184d42327ca488fb6d3ae6f561ec3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0448da810b044f5981a7da545f948f1f" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 42265 } } peers { permanent_uuid: "7184d42327ca488fb6d3ae6f561ec3ee" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 44807 } } peers { permanent_uuid: "23f4e90639fe4e4eb1e00cafb603c22b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 40499 } }
I20260929 20:34:11.165609 17106 leader_election.cc:288] T d2d9c58fa06b40299379a6787cf60d33 P 7184d42327ca488fb6d3ae6f561ec3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0448da810b044f5981a7da545f948f1f (127.28.244.131:42265), 23f4e90639fe4e4eb1e00cafb603c22b (127.28.244.129:40499)
I20260929 20:34:11.176026 17096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:11.166223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:51851 (local address 127.28.244.131:42265)
0929 20:34:11.166433 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:11.166443 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:11.167214 (+ 771us) server_negotiation.cc:407] Connection header received
0929 20:34:11.167458 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:11.167462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:11.167528 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:11.167641 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:34:11.168464 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:11.169542 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:11.173153 (+ 3611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:11.174451 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:11.174636 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:11.174718 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:11.175524 (+ 806us) server_negotiation.cc:299] Negotiation successful
0929 20:34:11.175676 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 20:34:11.176215 17109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:11.165939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:40499 (local address 127.28.244.130:33211)
0929 20:34:11.166973 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:11.166991 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:11.167129 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:11.167581 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:11.167586 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:11.167610 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:34:11.168065 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:11.168075 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:11.169573 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:11.169578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:11.172914 (+ 3336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:11.172936 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:11.174809 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:11.174817 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:11.175167 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:11.175343 (+ 176us) client_negotiation.cc:769] Sending connection context
0929 20:34:11.175455 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 20:34:11.175645 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":861,"mutex_wait_us":461,"spinlock_wait_cycles":26880,"thread_start_us":556,"threads_started":1}
I20260929 20:34:11.176239 17110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:11.166134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:42265 (local address 127.28.244.130:51851)
0929 20:34:11.167041 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:11.167091 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:11.167286 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:11.167656 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:11.167666 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:11.167788 (+ 122us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:34:11.168260 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:11.168270 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:11.169692 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:11.169701 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:11.172913 (+ 3212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:11.172936 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:11.174813 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:11.174818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:11.175165 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:11.175343 (+ 178us) client_negotiation.cc:769] Sending connection context
0929 20:34:11.175455 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 20:34:11.175645 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"mutex_wait_us":364,"spinlock_wait_cycles":37248,"thread_start_us":551,"threads_started":1}
I20260929 20:34:11.176072 17099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:11.166253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:33211 (local address 127.28.244.129:40499)
0929 20:34:11.166500 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:11.166509 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:11.167112 (+ 603us) server_negotiation.cc:407] Connection header received
0929 20:34:11.167394 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:11.167399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:11.167461 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:11.167564 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:34:11.168236 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:11.169411 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:11.173150 (+ 3739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:11.174448 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:11.174636 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:11.174718 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:11.175524 (+ 806us) server_negotiation.cc:299] Negotiation successful
0929 20:34:11.175676 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 20:34:11.177595 17036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d9c58fa06b40299379a6787cf60d33" candidate_uuid: "7184d42327ca488fb6d3ae6f561ec3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0448da810b044f5981a7da545f948f1f" is_pre_election: true
I20260929 20:34:11.177595 16764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d9c58fa06b40299379a6787cf60d33" candidate_uuid: "7184d42327ca488fb6d3ae6f561ec3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23f4e90639fe4e4eb1e00cafb603c22b" is_pre_election: true
I20260929 20:34:11.178246 16764 raft_consensus.cc:2454] T d2d9c58fa06b40299379a6787cf60d33 P 23f4e90639fe4e4eb1e00cafb603c22b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7184d42327ca488fb6d3ae6f561ec3ee in term 0.
I20260929 20:34:11.178246 17036 raft_consensus.cc:2454] T d2d9c58fa06b40299379a6787cf60d33 P 0448da810b044f5981a7da545f948f1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7184d42327ca488fb6d3ae6f561ec3ee in term 0.
I20260929 20:34:11.179095 16833 leader_election.cc:302] T d2d9c58fa06b40299379a6787cf60d33 P 7184d42327ca488fb6d3ae6f561ec3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0448da810b044f5981a7da545f948f1f, 7184d42327ca488fb6d3ae6f561ec3ee; no voters:
I20260929 20:34:11.179535 17106 raft_consensus.cc:2781] T d2d9c58fa06b40299379a6787cf60d33 P 7184d42327ca488fb6d3ae6f561ec3ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:34:11.179698 17106 raft_consensus.cc:482] T d2d9c58fa06b40299379a6787cf60d33 P 7184d42327ca488fb6d3ae6f561ec3ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:34:11.179827 17106 raft_consensus.cc:3037] T d2d9c58fa06b40299379a6787cf60d33 P 7184d42327ca488fb6d3ae6f561ec3ee [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:34:11.182042 17106 raft_consensus.cc:504] T d2d9c58fa06b40299379a6787cf60d33 P 7184d42327ca488fb6d3ae6f561ec3ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0448da810b044f5981a7da545f948f1f" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 42265 } } peers { permanent_uuid: "7184d42327ca488fb6d3ae6f561ec3ee" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 44807 } } peers { permanent_uuid: "23f4e90639fe4e4eb1e00cafb603c22b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 40499 } }
I20260929 20:34:11.182776 17106 leader_election.cc:288] T d2d9c58fa06b40299379a6787cf60d33 P 7184d42327ca488fb6d3ae6f561ec3ee [CANDIDATE]: Term 1 election: Requested vote from peers 0448da810b044f5981a7da545f948f1f (127.28.244.131:42265), 23f4e90639fe4e4eb1e00cafb603c22b (127.28.244.129:40499)
I20260929 20:34:11.183298 16764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d9c58fa06b40299379a6787cf60d33" candidate_uuid: "7184d42327ca488fb6d3ae6f561ec3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23f4e90639fe4e4eb1e00cafb603c22b"
I20260929 20:34:11.183326 17036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d9c58fa06b40299379a6787cf60d33" candidate_uuid: "7184d42327ca488fb6d3ae6f561ec3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0448da810b044f5981a7da545f948f1f"
I20260929 20:34:11.183537 16764 raft_consensus.cc:3037] T d2d9c58fa06b40299379a6787cf60d33 P 23f4e90639fe4e4eb1e00cafb603c22b [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:34:11.183550 17036 raft_consensus.cc:3037] T d2d9c58fa06b40299379a6787cf60d33 P 0448da810b044f5981a7da545f948f1f [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:34:11.186923 16764 raft_consensus.cc:2454] T d2d9c58fa06b40299379a6787cf60d33 P 23f4e90639fe4e4eb1e00cafb603c22b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7184d42327ca488fb6d3ae6f561ec3ee in term 1.
I20260929 20:34:11.186923 17036 raft_consensus.cc:2454] T d2d9c58fa06b40299379a6787cf60d33 P 0448da810b044f5981a7da545f948f1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7184d42327ca488fb6d3ae6f561ec3ee in term 1.
I20260929 20:34:11.187384 16835 leader_election.cc:302] T d2d9c58fa06b40299379a6787cf60d33 P 7184d42327ca488fb6d3ae6f561ec3ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23f4e90639fe4e4eb1e00cafb603c22b, 7184d42327ca488fb6d3ae6f561ec3ee; no voters:
I20260929 20:34:11.187709 17106 raft_consensus.cc:2781] T d2d9c58fa06b40299379a6787cf60d33 P 7184d42327ca488fb6d3ae6f561ec3ee [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:34:11.188249 17106 raft_consensus.cc:686] T d2d9c58fa06b40299379a6787cf60d33 P 7184d42327ca488fb6d3ae6f561ec3ee [term 1 LEADER]: Becoming Leader. State: Replica: 7184d42327ca488fb6d3ae6f561ec3ee, State: Running, Role: LEADER
I20260929 20:34:11.188858 17106 consensus_queue.cc:237] T d2d9c58fa06b40299379a6787cf60d33 P 7184d42327ca488fb6d3ae6f561ec3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0448da810b044f5981a7da545f948f1f" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 42265 } } peers { permanent_uuid: "7184d42327ca488fb6d3ae6f561ec3ee" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 44807 } } peers { permanent_uuid: "23f4e90639fe4e4eb1e00cafb603c22b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 40499 } }
I20260929 20:34:11.193791 16618 catalog_manager.cc:5495] T d2d9c58fa06b40299379a6787cf60d33 P 7184d42327ca488fb6d3ae6f561ec3ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 7184d42327ca488fb6d3ae6f561ec3ee (127.28.244.130). New cstate: current_term: 1 leader_uuid: "7184d42327ca488fb6d3ae6f561ec3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0448da810b044f5981a7da545f948f1f" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 42265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7184d42327ca488fb6d3ae6f561ec3ee" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 44807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23f4e90639fe4e4eb1e00cafb603c22b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 40499 } health_report { overall_health: UNKNOWN } } }
I20260929 20:34:11.240075 17097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:11.233672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33750 (local address 127.28.244.130:44807)
0929 20:34:11.233938 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:11.233944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:11.233972 (+ 28us) server_negotiation.cc:407] Connection header received
0929 20:34:11.234195 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:11.234199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:11.234300 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:11.234454 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:34:11.234879 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:11.235686 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:11.237409 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:11.237906 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:11.238024 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:11.238609 (+ 585us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:34:11.239209 (+ 600us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:34:11.239343 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:11.239549 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 20:34:11.239638 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260929 20:34:11.249480 17036 raft_consensus.cc:1261] T d2d9c58fa06b40299379a6787cf60d33 P 0448da810b044f5981a7da545f948f1f [term 1 FOLLOWER]: Refusing update from remote peer 7184d42327ca488fb6d3ae6f561ec3ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:34:11.249480 16764 raft_consensus.cc:1261] T d2d9c58fa06b40299379a6787cf60d33 P 23f4e90639fe4e4eb1e00cafb603c22b [term 1 FOLLOWER]: Refusing update from remote peer 7184d42327ca488fb6d3ae6f561ec3ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:34:11.250626 17106 consensus_queue.cc:1035] T d2d9c58fa06b40299379a6787cf60d33 P 7184d42327ca488fb6d3ae6f561ec3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "23f4e90639fe4e4eb1e00cafb603c22b" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 40499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:34:11.250999 17111 consensus_queue.cc:1035] T d2d9c58fa06b40299379a6787cf60d33 P 7184d42327ca488fb6d3ae6f561ec3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0448da810b044f5981a7da545f948f1f" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 42265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:34:11.264572 17114 mvcc.cc:205] Tried to move back new op lower bound from 7334764753904709632 to 7334764753673928704. Current Snapshot: MvccSnapshot[applied={T|T < 7334764753904709632}]
I20260929 20:34:11.280227 16618 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:34:11.280522 16618 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:34:11.284420 16618 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7373441ce3141e3a200b243eeca2723: Sending DeleteTablet for 3 replicas of tablet d2d9c58fa06b40299379a6787cf60d33
I20260929 20:34:11.285753 17016 tablet_service.cc:1514] Processing DeleteTablet for tablet d2d9c58fa06b40299379a6787cf60d33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:34:11 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:55230
I20260929 20:34:11.285989 16744 tablet_service.cc:1514] Processing DeleteTablet for tablet d2d9c58fa06b40299379a6787cf60d33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:34:11 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:52688
I20260929 20:34:11.286000 16880 tablet_service.cc:1514] Processing DeleteTablet for tablet d2d9c58fa06b40299379a6787cf60d33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:34:11 UTC) from {username='kudu', principal='kudu/127.28.244.190@KRBTEST.COM'} at 127.0.0.1:33740
I20260929 20:34:11.288427 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 16681
I20260929 20:34:11.297354 17124 tablet_replica.cc:330] T d2d9c58fa06b40299379a6787cf60d33 P 0448da810b044f5981a7da545f948f1f: stopping tablet replica
I20260929 20:34:11.298060 17124 raft_consensus.cc:2229] T d2d9c58fa06b40299379a6787cf60d33 P 0448da810b044f5981a7da545f948f1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:34:11.298774 17124 raft_consensus.cc:2258] T d2d9c58fa06b40299379a6787cf60d33 P 0448da810b044f5981a7da545f948f1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:34:11.300343 17125 tablet_replica.cc:330] T d2d9c58fa06b40299379a6787cf60d33 P 7184d42327ca488fb6d3ae6f561ec3ee: stopping tablet replica
I20260929 20:34:11.300822 17125 raft_consensus.cc:2229] T d2d9c58fa06b40299379a6787cf60d33 P 7184d42327ca488fb6d3ae6f561ec3ee [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:34:11.301519 17125 raft_consensus.cc:2258] T d2d9c58fa06b40299379a6787cf60d33 P 7184d42327ca488fb6d3ae6f561ec3ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:34:11.301548 17124 ts_tablet_manager.cc:1905] T d2d9c58fa06b40299379a6787cf60d33 P 0448da810b044f5981a7da545f948f1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:34:11.302345 17127 tablet_replica.cc:330] T d2d9c58fa06b40299379a6787cf60d33 P 23f4e90639fe4e4eb1e00cafb603c22b: stopping tablet replica
I20260929 20:34:11.302970 17127 raft_consensus.cc:2229] T d2d9c58fa06b40299379a6787cf60d33 P 23f4e90639fe4e4eb1e00cafb603c22b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:34:11.303567 17127 raft_consensus.cc:2258] T d2d9c58fa06b40299379a6787cf60d33 P 23f4e90639fe4e4eb1e00cafb603c22b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:34:11.306151 17127 ts_tablet_manager.cc:1905] T d2d9c58fa06b40299379a6787cf60d33 P 23f4e90639fe4e4eb1e00cafb603c22b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:34:11.309229 17124 ts_tablet_manager.cc:1918] T d2d9c58fa06b40299379a6787cf60d33 P 0448da810b044f5981a7da545f948f1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:34:11.309424 17124 log.cc:1196] T d2d9c58fa06b40299379a6787cf60d33 P 0448da810b044f5981a7da545f948f1f: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-2/wal/wals/d2d9c58fa06b40299379a6787cf60d33
Sep 29 20:34:11 dist-test-slave-p29l krb5kdc[16574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.129@KRBTEST.COM
I20260929 20:34:11.310007 17124 ts_tablet_manager.cc:1939] T d2d9c58fa06b40299379a6787cf60d33 P 0448da810b044f5981a7da545f948f1f: Deleting consensus metadata
I20260929 20:34:11.312288 16604 catalog_manager.cc:4841] TS 0448da810b044f5981a7da545f948f1f (127.28.244.131:42265): tablet d2d9c58fa06b40299379a6787cf60d33 (table test-table [id=f4e7e6872e5f47d7a0a0d847f343c99a]) successfully deleted
I20260929 20:34:11.313266 17125 ts_tablet_manager.cc:1905] T d2d9c58fa06b40299379a6787cf60d33 P 7184d42327ca488fb6d3ae6f561ec3ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:34:11.317793 17127 ts_tablet_manager.cc:1918] T d2d9c58fa06b40299379a6787cf60d33 P 23f4e90639fe4e4eb1e00cafb603c22b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:34:11.317978 17127 log.cc:1196] T d2d9c58fa06b40299379a6787cf60d33 P 23f4e90639fe4e4eb1e00cafb603c22b: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-0/wal/wals/d2d9c58fa06b40299379a6787cf60d33
I20260929 20:34:11.318532 17127 ts_tablet_manager.cc:1939] T d2d9c58fa06b40299379a6787cf60d33 P 23f4e90639fe4e4eb1e00cafb603c22b: Deleting consensus metadata
I20260929 20:34:11.319463 17099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:11.290144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52692 (local address 127.28.244.129:40499)
0929 20:34:11.290392 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:11.290401 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:11.295225 (+ 4824us) server_negotiation.cc:407] Connection header received
0929 20:34:11.296708 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:11.296718 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:11.296844 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:11.296999 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:11.303055 (+ 6056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:11.304247 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:11.305728 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:11.306184 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:11.306341 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:11.311041 (+ 4700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:11.311057 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:11.311073 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:11.311112 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:11.315087 (+ 3975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:11.316208 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:11.316219 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:11.316222 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:11.316443 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:11.317002 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:11.317007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:11.317010 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:11.317530 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:11.317870 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:11.318751 (+ 881us) server_negotiation.cc:299] Negotiation successful
0929 20:34:11.319021 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 20:34:11.320461 17125 ts_tablet_manager.cc:1918] T d2d9c58fa06b40299379a6787cf60d33 P 7184d42327ca488fb6d3ae6f561ec3ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:34:11.320643 17125 log.cc:1196] T d2d9c58fa06b40299379a6787cf60d33 P 7184d42327ca488fb6d3ae6f561ec3ee: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713721201454-29650-0/minicluster-data/ts-1/wal/wals/d2d9c58fa06b40299379a6787cf60d33
I20260929 20:34:11.320644 16606 catalog_manager.cc:4841] TS 23f4e90639fe4e4eb1e00cafb603c22b (127.28.244.129:40499): tablet d2d9c58fa06b40299379a6787cf60d33 (table test-table [id=f4e7e6872e5f47d7a0a0d847f343c99a]) successfully deleted
I20260929 20:34:11.321233 17125 ts_tablet_manager.cc:1939] T d2d9c58fa06b40299379a6787cf60d33 P 7184d42327ca488fb6d3ae6f561ec3ee: Deleting consensus metadata
I20260929 20:34:11.321481 16804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:34:11.323423 16603 catalog_manager.cc:4841] TS 7184d42327ca488fb6d3ae6f561ec3ee (127.28.244.130:44807): tablet d2d9c58fa06b40299379a6787cf60d33 (table test-table [id=f4e7e6872e5f47d7a0a0d847f343c99a]) successfully deleted
I20260929 20:34:11.428727 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 16681
I20260929 20:34:11.449338 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 16816
Sep 29 20:34:11 dist-test-slave-p29l krb5kdc[16574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.130@KRBTEST.COM
I20260929 20:34:11.468091 17097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:11.450124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33754 (local address 127.28.244.130:44807)
0929 20:34:11.450301 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:11.450305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:11.450330 (+ 25us) server_negotiation.cc:407] Connection header received
0929 20:34:11.450548 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:11.450553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:11.450629 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:11.450714 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:11.452434 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:11.453285 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:11.455198 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:11.455638 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:11.455728 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:11.461481 (+ 5753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:11.461498 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:11.461515 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:11.461561 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:11.464967 (+ 3406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:11.465928 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:11.465936 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:11.465938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:11.466105 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:11.466523 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:11.466527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:11.466529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:11.466893 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:11.467166 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:11.467504 (+ 338us) server_negotiation.cc:299] Negotiation successful
0929 20:34:11.467684 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:34:11.468875 16941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:34:11.585112 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 16816
I20260929 20:34:11.607404 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 16952
Sep 29 20:34:11 dist-test-slave-p29l krb5kdc[16574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.131@KRBTEST.COM
I20260929 20:34:11.623973 17096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:11.608286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55240 (local address 127.28.244.131:42265)
0929 20:34:11.608505 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:11.608513 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:11.608537 (+ 24us) server_negotiation.cc:407] Connection header received
0929 20:34:11.608761 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:11.608767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:11.608894 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:11.609008 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:11.610396 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:11.611166 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:11.612725 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:11.613153 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:11.613296 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:11.618118 (+ 4822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:11.618140 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:11.618157 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:11.618215 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:11.620952 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:11.621720 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:11.621728 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:11.621731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:11.621896 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:11.622314 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:11.622319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:11.622323 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:11.622702 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:11.623030 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:11.623369 (+ 339us) server_negotiation.cc:299] Negotiation successful
0929 20:34:11.623593 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 20:34:11.624712 17076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:34:11.716583 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 16952
I20260929 20:34:11.734200 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 16587
I20260929 20:34:11.735301 16650 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:34:11.847735 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 16587
2026-09-29T20:34:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4623 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T20:34:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:34:11Z Disabled control of system clock
I20260929 20:34:11.930274 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:42259
--webserver_interface=127.28.244.190
--webserver_port=0
--builtin_ntp_servers=127.28.244.148:38919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:42259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 20:34:12.293849 17136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:34:12.294190 17136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:34:12.294276 17136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:34:12.301927 17136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:34:12.302059 17136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:34:12.302098 17136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:34:12.302150 17136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:34:12.302203 17136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:34:12.311913 17136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:38919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:42259
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:42259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:34:12.313300 17136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:34:12.315167 17136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:34:12.327840 17141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:12.327920 17142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:12.327877 17144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:34:12.328691 17136 server_base.cc:1034] running on GCE node
I20260929 20:34:12.329586 17136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:34:12.331161 17136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:34:12.332433 17136 hybrid_clock.cc:648] HybridClock initialized: now 1790714052332380 us; error 75 us; skew 500 ppm
I20260929 20:34:12.332836 17136 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:34:12.340441 17136 webserver.cc:460] Webserver started at http://127.28.244.190:38973/ using document root <none> and password file <none>
I20260929 20:34:12.341300 17136 fs_manager.cc:359] Metadata directory not provided
I20260929 20:34:12.341462 17136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:34:12.341830 17136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:34:12.345240 17136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "a71a80705b314928b1412254b8dc672e"
format_stamp: "Formatted at 2026-09-29 20:34:12 on dist-test-slave-p29l"
server_key: "f8f5ed4f3d95a28921b750ae2c04eb8d"
server_key_iv: "32717ba9d74a932f9d59916cda03ffa3"
server_key_version: "encryptionkey@0"
I20260929 20:34:12.346177 17136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "a71a80705b314928b1412254b8dc672e"
format_stamp: "Formatted at 2026-09-29 20:34:12 on dist-test-slave-p29l"
server_key: "f8f5ed4f3d95a28921b750ae2c04eb8d"
server_key_iv: "32717ba9d74a932f9d59916cda03ffa3"
server_key_version: "encryptionkey@0"
I20260929 20:34:12.352303 17136 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 20:34:12.356431 17150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:12.358080 17136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:34:12.358314 17136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "a71a80705b314928b1412254b8dc672e"
format_stamp: "Formatted at 2026-09-29 20:34:12 on dist-test-slave-p29l"
server_key: "f8f5ed4f3d95a28921b750ae2c04eb8d"
server_key_iv: "32717ba9d74a932f9d59916cda03ffa3"
server_key_version: "encryptionkey@0"
I20260929 20:34:12.358570 17136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:34:12.380785 17136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:34:12.381762 17136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:34:12.382107 17136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:34:12.402218 17136 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:42259
I20260929 20:34:12.402238 17201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:42259 every 8 connection(s)
I20260929 20:34:12.403728 17136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:34:12.407939 17202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:34:12.408036 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 17136
I20260929 20:34:12.408212 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:34:12.408674 29650 external_mini_cluster.cc:1428] Setting key d2dfc76517bf88a30b9d7a84062ec1a7
I20260929 20:34:12.424926 17202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a71a80705b314928b1412254b8dc672e: Bootstrap starting.
I20260929 20:34:12.425254 17205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:12.410725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55472 (local address 127.28.244.190:42259)
0929 20:34:12.411980 (+ 1255us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:12.411997 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:12.412054 (+ 57us) server_negotiation.cc:407] Connection header received
0929 20:34:12.414248 (+ 2194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:12.414285 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:12.415001 (+ 716us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:12.415491 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:12.416147 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:12.418230 (+ 2083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:12.419621 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:12.420757 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:12.420973 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:12.422116 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:12.422127 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:12.422151 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:34:12.422939 (+ 788us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:12.423044 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:34:12.423076 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:12.423193 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:12.423462 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 20:34:12.423871 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":399,"threads_started":1}
I20260929 20:34:12.428889 17202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a71a80705b314928b1412254b8dc672e: Neither blocks nor log segments found. Creating new log.
I20260929 20:34:12.429883 17202 log.cc:824] T 00000000000000000000000000000000 P a71a80705b314928b1412254b8dc672e: Log is configured to *not* fsync() on all Append() calls
I20260929 20:34:12.432890 17202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a71a80705b314928b1412254b8dc672e: No bootstrap required, opened a new log
I20260929 20:34:12.440519 17202 raft_consensus.cc:348] T 00000000000000000000000000000000 P a71a80705b314928b1412254b8dc672e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a71a80705b314928b1412254b8dc672e" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 42259 } }
I20260929 20:34:12.440980 17202 raft_consensus.cc:374] T 00000000000000000000000000000000 P a71a80705b314928b1412254b8dc672e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:34:12.441116 17202 raft_consensus.cc:729] T 00000000000000000000000000000000 P a71a80705b314928b1412254b8dc672e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a71a80705b314928b1412254b8dc672e, State: Initialized, Role: FOLLOWER
I20260929 20:34:12.441731 17202 consensus_queue.cc:260] T 00000000000000000000000000000000 P a71a80705b314928b1412254b8dc672e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a71a80705b314928b1412254b8dc672e" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 42259 } }
I20260929 20:34:12.441985 17202 raft_consensus.cc:388] T 00000000000000000000000000000000 P a71a80705b314928b1412254b8dc672e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:34:12.442129 17202 raft_consensus.cc:482] T 00000000000000000000000000000000 P a71a80705b314928b1412254b8dc672e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:34:12.442308 17202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a71a80705b314928b1412254b8dc672e [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:34:12.444696 17202 raft_consensus.cc:504] T 00000000000000000000000000000000 P a71a80705b314928b1412254b8dc672e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a71a80705b314928b1412254b8dc672e" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 42259 } }
I20260929 20:34:12.445268 17202 leader_election.cc:302] T 00000000000000000000000000000000 P a71a80705b314928b1412254b8dc672e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a71a80705b314928b1412254b8dc672e; no voters:
I20260929 20:34:12.445884 17202 leader_election.cc:288] T 00000000000000000000000000000000 P a71a80705b314928b1412254b8dc672e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:34:12.446070 17207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a71a80705b314928b1412254b8dc672e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:34:12.446815 17207 raft_consensus.cc:686] T 00000000000000000000000000000000 P a71a80705b314928b1412254b8dc672e [term 1 LEADER]: Becoming Leader. State: Replica: a71a80705b314928b1412254b8dc672e, State: Running, Role: LEADER
I20260929 20:34:12.447386 17207 consensus_queue.cc:237] T 00000000000000000000000000000000 P a71a80705b314928b1412254b8dc672e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a71a80705b314928b1412254b8dc672e" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 42259 } }
I20260929 20:34:12.448215 17202 sys_catalog.cc:564] T 00000000000000000000000000000000 P a71a80705b314928b1412254b8dc672e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:34:12.450361 17209 sys_catalog.cc:455] T 00000000000000000000000000000000 P a71a80705b314928b1412254b8dc672e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a71a80705b314928b1412254b8dc672e. Latest consensus state: current_term: 1 leader_uuid: "a71a80705b314928b1412254b8dc672e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a71a80705b314928b1412254b8dc672e" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 42259 } } }
I20260929 20:34:12.450306 17208 sys_catalog.cc:455] T 00000000000000000000000000000000 P a71a80705b314928b1412254b8dc672e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a71a80705b314928b1412254b8dc672e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a71a80705b314928b1412254b8dc672e" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 42259 } } }
I20260929 20:34:12.450804 17209 sys_catalog.cc:458] T 00000000000000000000000000000000 P a71a80705b314928b1412254b8dc672e [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:12.450825 17208 sys_catalog.cc:458] T 00000000000000000000000000000000 P a71a80705b314928b1412254b8dc672e [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:12.456202 17214 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:34:12.461972 17214 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:34:12.472540 17214 catalog_manager.cc:1347] Generated new cluster ID: ffad1cb0b043430a8f7ba9665b092041
I20260929 20:34:12.472668 17214 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:34:12.499905 17214 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:34:12.501323 17214 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:34:12.507752 17214 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a71a80705b314928b1412254b8dc672e: Generated new TSK 0
I20260929 20:34:12.508965 17214 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:34:12.524933 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.129:0
--local_ip_for_outbound_sockets=127.28.244.129
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:42259
--builtin_ntp_servers=127.28.244.148:38919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 20:34:12.884222 17226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:34:12.884570 17226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:34:12.884742 17226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:34:12.892432 17226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:34:12.892588 17226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:34:12.892793 17226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.129
I20260929 20:34:12.902245 17226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:38919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.244.129
--webserver_port=0
--tserver_master_addrs=127.28.244.190:42259
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.129
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:34:12.903662 17226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:34:12.905577 17226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:34:12.920110 17231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:12.920171 17234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:12.920149 17232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:34:12.920765 17226 server_base.cc:1034] running on GCE node
I20260929 20:34:12.921464 17226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:34:12.922560 17226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:34:12.923894 17226 hybrid_clock.cc:648] HybridClock initialized: now 1790714052923786 us; error 38 us; skew 500 ppm
I20260929 20:34:12.924288 17226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:34:12.926927 17226 webserver.cc:460] Webserver started at http://127.28.244.129:45835/ using document root <none> and password file <none>
I20260929 20:34:12.927721 17226 fs_manager.cc:359] Metadata directory not provided
I20260929 20:34:12.927891 17226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:34:12.928231 17226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:34:12.931337 17226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-0/data/instance:
uuid: "194ecaaf435f407fa978f171bc22c86c"
format_stamp: "Formatted at 2026-09-29 20:34:12 on dist-test-slave-p29l"
server_key: "0aef6b069f1f633c418763611e1c7e52"
server_key_iv: "c3cd3fcbcd85253d78fd0fa6a095372d"
server_key_version: "encryptionkey@0"
I20260929 20:34:12.932286 17226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance:
uuid: "194ecaaf435f407fa978f171bc22c86c"
format_stamp: "Formatted at 2026-09-29 20:34:12 on dist-test-slave-p29l"
server_key: "0aef6b069f1f633c418763611e1c7e52"
server_key_iv: "c3cd3fcbcd85253d78fd0fa6a095372d"
server_key_version: "encryptionkey@0"
I20260929 20:34:12.938261 17226 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 20:34:12.942298 17240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:12.943862 17226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:34:12.944063 17226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-0/wal
uuid: "194ecaaf435f407fa978f171bc22c86c"
format_stamp: "Formatted at 2026-09-29 20:34:12 on dist-test-slave-p29l"
server_key: "0aef6b069f1f633c418763611e1c7e52"
server_key_iv: "c3cd3fcbcd85253d78fd0fa6a095372d"
server_key_version: "encryptionkey@0"
I20260929 20:34:12.944308 17226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:34:12.967630 17226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:34:12.968586 17226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:34:12.968924 17226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:34:12.969947 17226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:34:12.971529 17226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:34:12.971661 17226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:12.971791 17226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:34:12.971943 17226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260929 20:34:13.006798 17226 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.129:36123
I20260929 20:34:13.006916 17353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.129:36123 every 8 connection(s)
I20260929 20:34:13.008258 17226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-0/data/info.pb
I20260929 20:34:13.015625 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 17226
I20260929 20:34:13.015853 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-0/wal/instance
I20260929 20:34:13.016325 29650 external_mini_cluster.cc:1428] Setting key 20c5412cb53549166bad494b34365478
I20260929 20:34:13.019491 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.130:0
--local_ip_for_outbound_sockets=127.28.244.130
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:42259
--builtin_ntp_servers=127.28.244.148:38919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 20:34:13.022965 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:13.011179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.129:59885 (local address 127.28.244.190:42259)
0929 20:34:13.011684 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:13.011689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:13.012951 (+ 1262us) server_negotiation.cc:407] Connection header received
0929 20:34:13.014822 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:13.014829 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:13.014930 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:13.015056 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:13.016370 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:13.017313 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:13.019706 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:13.020214 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:13.020328 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:13.022145 (+ 1817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:13.022149 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:13.022153 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:34:13.022172 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:13.022222 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:34:13.022235 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:13.022311 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:13.022606 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 20:34:13.022698 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":296,"threads_started":1}
I20260929 20:34:13.024567 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:13.011734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:42259 (local address 127.28.244.129:59885)
0929 20:34:13.012717 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:13.012778 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:13.014525 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:13.015325 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:13.015334 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:13.015429 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:34:13.016174 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:13.016195 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:13.017529 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:13.017533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:13.019487 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:13.019505 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:13.020481 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:13.020491 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:13.020899 (+ 408us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:13.021714 (+ 815us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:34:13.021720 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:34:13.021781 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:34:13.021907 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:34:13.022350 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:34:13.022365 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:34:13.022377 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 20:34:13.022581 (+ 204us) client_negotiation.cc:240] Negotiation successful
0929 20:34:13.023060 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":274,"threads_started":1}
I20260929 20:34:13.026312 17354 heartbeater.cc:347] Connected to a master server at 127.28.244.190:42259
I20260929 20:34:13.026650 17354 heartbeater.cc:464] Registering TS with master...
I20260929 20:34:13.027495 17354 heartbeater.cc:511] Master 127.28.244.190:42259 requested a full tablet report, sending...
I20260929 20:34:13.030272 17167 ts_manager.cc:194] Registered new tserver with Master: 194ecaaf435f407fa978f171bc22c86c (127.28.244.129:36123)
I20260929 20:34:13.031996 17167 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.244.129:59885
W20260929 20:34:13.369458 17358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:34:13.369827 17358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:34:13.369997 17358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:34:13.377403 17358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:34:13.377565 17358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:34:13.377743 17358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.130
I20260929 20:34:13.387208 17358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:38919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.244.130
--webserver_port=0
--tserver_master_addrs=127.28.244.190:42259
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.130
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:34:13.388640 17358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:34:13.390476 17358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:34:13.404867 17364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:13.405205 17363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:13.404909 17366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:34:13.406178 17358 server_base.cc:1034] running on GCE node
I20260929 20:34:13.406908 17358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:34:13.408190 17358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:34:13.409447 17358 hybrid_clock.cc:648] HybridClock initialized: now 1790714053409374 us; error 64 us; skew 500 ppm
I20260929 20:34:13.409865 17358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:34:13.412849 17358 webserver.cc:460] Webserver started at http://127.28.244.130:43239/ using document root <none> and password file <none>
I20260929 20:34:13.413698 17358 fs_manager.cc:359] Metadata directory not provided
I20260929 20:34:13.413815 17358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:34:13.414193 17358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:34:13.418244 17358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-1/data/instance:
uuid: "e1dc00bec9be4e04b7bf9818ac33f624"
format_stamp: "Formatted at 2026-09-29 20:34:13 on dist-test-slave-p29l"
server_key: "fcd583e27779092862e26cd52791c8fd"
server_key_iv: "f5fe80f092a0aa54e9b5d017a6a66fee"
server_key_version: "encryptionkey@0"
I20260929 20:34:13.419302 17358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance:
uuid: "e1dc00bec9be4e04b7bf9818ac33f624"
format_stamp: "Formatted at 2026-09-29 20:34:13 on dist-test-slave-p29l"
server_key: "fcd583e27779092862e26cd52791c8fd"
server_key_iv: "f5fe80f092a0aa54e9b5d017a6a66fee"
server_key_version: "encryptionkey@0"
I20260929 20:34:13.425977 17358 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260929 20:34:13.429842 17372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:13.431346 17358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:34:13.431560 17358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-1/wal
uuid: "e1dc00bec9be4e04b7bf9818ac33f624"
format_stamp: "Formatted at 2026-09-29 20:34:13 on dist-test-slave-p29l"
server_key: "fcd583e27779092862e26cd52791c8fd"
server_key_iv: "f5fe80f092a0aa54e9b5d017a6a66fee"
server_key_version: "encryptionkey@0"
I20260929 20:34:13.431797 17358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:34:13.448060 17358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:34:13.449352 17358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:34:13.449721 17358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:34:13.450783 17358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:34:13.452445 17358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:34:13.452572 17358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:13.452708 17358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:34:13.452821 17358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:13.487257 17358 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.130:45457
I20260929 20:34:13.487310 17486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.130:45457 every 8 connection(s)
I20260929 20:34:13.488700 17358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-1/data/info.pb
I20260929 20:34:13.489208 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 17358
I20260929 20:34:13.489389 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-1/wal/instance
I20260929 20:34:13.490051 29650 external_mini_cluster.cc:1428] Setting key d6ffa9c85d53230248c846ff0dbbe2d7
I20260929 20:34:13.493630 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.244.131:0
--local_ip_for_outbound_sockets=127.28.244.131
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:42259
--builtin_ntp_servers=127.28.244.148:38919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 20:34:13.505209 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:13.491484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.130:48017 (local address 127.28.244.190:42259)
0929 20:34:13.491697 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:13.491703 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:13.493604 (+ 1901us) server_negotiation.cc:407] Connection header received
0929 20:34:13.496591 (+ 2987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:13.496595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:13.496672 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:13.496763 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:13.498370 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:13.499195 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:13.501976 (+ 2781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:13.502439 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:13.502559 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:13.504323 (+ 1764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:13.504329 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:13.504332 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:34:13.504354 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:13.504388 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:34:13.504401 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:13.504505 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:13.504752 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 20:34:13.504880 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 20:34:13.506327 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:13.492209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:42259 (local address 127.28.244.130:48017)
0929 20:34:13.493419 (+ 1210us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:13.493477 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:13.496255 (+ 2778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:13.497060 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:13.497075 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:13.497211 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:34:13.498089 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:13.498121 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:13.499393 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:13.499402 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:13.501749 (+ 2347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:13.501770 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:13.502708 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:13.502717 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:13.503097 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:13.503943 (+ 846us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:34:13.503945 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:34:13.503989 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:34:13.504086 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:34:13.504503 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:34:13.504516 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:34:13.504527 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 20:34:13.504726 (+ 199us) client_negotiation.cc:240] Negotiation successful
0929 20:34:13.505173 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":337,"threads_started":1}
I20260929 20:34:13.508064 17487 heartbeater.cc:347] Connected to a master server at 127.28.244.190:42259
I20260929 20:34:13.508430 17487 heartbeater.cc:464] Registering TS with master...
I20260929 20:34:13.509234 17487 heartbeater.cc:511] Master 127.28.244.190:42259 requested a full tablet report, sending...
I20260929 20:34:13.510819 17167 ts_manager.cc:194] Registered new tserver with Master: e1dc00bec9be4e04b7bf9818ac33f624 (127.28.244.130:45457)
I20260929 20:34:13.511658 17167 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.244.130:48017
W20260929 20:34:13.844646 17490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:34:13.845009 17490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:34:13.845173 17490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:34:13.853066 17490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:34:13.853220 17490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:34:13.853404 17490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.244.131
I20260929 20:34:13.863428 17490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:38919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.244.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.244.131
--webserver_port=0
--tserver_master_addrs=127.28.244.190:42259
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.244.131
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:34:13.864964 17490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:34:13.866850 17490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:34:13.880582 17496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:13.880756 17498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:13.881103 17495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:34:13.881562 17490 server_base.cc:1034] running on GCE node
I20260929 20:34:13.882221 17490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:34:13.883234 17490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:34:13.884449 17490 hybrid_clock.cc:648] HybridClock initialized: now 1790714053884400 us; error 33 us; skew 500 ppm
I20260929 20:34:13.884800 17490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:34:13.887315 17490 webserver.cc:460] Webserver started at http://127.28.244.131:37959/ using document root <none> and password file <none>
I20260929 20:34:13.888170 17490 fs_manager.cc:359] Metadata directory not provided
I20260929 20:34:13.888262 17490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:34:13.888540 17490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:34:13.891482 17490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-2/data/instance:
uuid: "f127c072fea84fc89b26fb049a560d8a"
format_stamp: "Formatted at 2026-09-29 20:34:13 on dist-test-slave-p29l"
server_key: "d73b20efa0088da5b73ab9f50936dd13"
server_key_iv: "ac47d4f52a100345ec2be42db54e1991"
server_key_version: "encryptionkey@0"
I20260929 20:34:13.892333 17490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance:
uuid: "f127c072fea84fc89b26fb049a560d8a"
format_stamp: "Formatted at 2026-09-29 20:34:13 on dist-test-slave-p29l"
server_key: "d73b20efa0088da5b73ab9f50936dd13"
server_key_iv: "ac47d4f52a100345ec2be42db54e1991"
server_key_version: "encryptionkey@0"
I20260929 20:34:13.897838 17490 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:34:13.901394 17504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:13.902743 17490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:34:13.902925 17490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-2/wal
uuid: "f127c072fea84fc89b26fb049a560d8a"
format_stamp: "Formatted at 2026-09-29 20:34:13 on dist-test-slave-p29l"
server_key: "d73b20efa0088da5b73ab9f50936dd13"
server_key_iv: "ac47d4f52a100345ec2be42db54e1991"
server_key_version: "encryptionkey@0"
I20260929 20:34:13.903152 17490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:34:13.915573 17490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:34:13.916646 17490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:34:13.916954 17490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:34:13.917943 17490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:34:13.919476 17490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:34:13.919601 17490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:13.919742 17490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:34:13.919905 17490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:13.955246 17490 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.131:33689
I20260929 20:34:13.955302 17618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.131:33689 every 8 connection(s)
I20260929 20:34:13.956759 17490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-2/data/info.pb
I20260929 20:34:13.963996 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 17490
I20260929 20:34:13.964191 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-2/wal/instance
I20260929 20:34:13.964756 29650 external_mini_cluster.cc:1428] Setting key fd110ac58a22a78f9d1093df231cf739
I20260929 20:34:13.970022 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:13.959515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:50993 (local address 127.28.244.190:42259)
0929 20:34:13.959756 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:13.959762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:13.961241 (+ 1479us) server_negotiation.cc:407] Connection header received
0929 20:34:13.963059 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:13.963063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:13.963138 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:13.963230 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:13.964752 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:13.965610 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:13.967232 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:13.967629 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:13.967765 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:13.969148 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:13.969153 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:13.969156 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:34:13.969174 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:13.969207 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:34:13.969219 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:13.969318 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:13.969579 (+ 261us) server_negotiation.cc:299] Negotiation successful
0929 20:34:13.969686 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 20:34:13.971102 17621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:13.960126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:42259 (local address 127.28.244.131:50993)
0929 20:34:13.961072 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:13.961118 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:13.962785 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:13.963440 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:13.963449 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:13.963588 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:34:13.964537 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:13.964566 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:13.965773 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:13.965779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:13.967073 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:13.967086 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:13.967902 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:13.967910 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:13.968148 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:13.968807 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:34:13.968810 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:34:13.968857 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:34:13.968939 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:34:13.969317 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:34:13.969330 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:34:13.969367 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 20:34:13.969552 (+ 185us) client_negotiation.cc:240] Negotiation successful
0929 20:34:13.969976 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":299,"threads_started":1}
I20260929 20:34:13.972553 17619 heartbeater.cc:347] Connected to a master server at 127.28.244.190:42259
I20260929 20:34:13.972877 17619 heartbeater.cc:464] Registering TS with master...
I20260929 20:34:13.973613 17619 heartbeater.cc:511] Master 127.28.244.190:42259 requested a full tablet report, sending...
I20260929 20:34:13.975085 17167 ts_manager.cc:194] Registered new tserver with Master: f127c072fea84fc89b26fb049a560d8a (127.28.244.131:33689)
I20260929 20:34:13.975939 17167 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.244.131:50993
I20260929 20:34:13.979993 29650 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:34:13.990336 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:13.984216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55482 (local address 127.28.244.190:42259)
0929 20:34:13.984455 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:13.984460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:13.984933 (+ 473us) server_negotiation.cc:407] Connection header received
0929 20:34:13.985088 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:13.985092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:13.985164 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:13.985282 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:13.985717 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:13.986512 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:13.987914 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:13.988333 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:13.988450 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:13.989627 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:13.989631 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:13.989635 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:34:13.989654 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:13.989687 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:34:13.989699 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:13.989841 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:13.990000 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 20:34:13.990089 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 20:34:13.994490 17167 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:34:14.001740 17167 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:34:14.035233 17354 heartbeater.cc:503] Master 127.28.244.190:42259 was elected leader, sending a full tablet report...
I20260929 20:34:14.036237 17632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:14.015351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:36123 (local address 127.0.0.1:46822)
0929 20:34:14.026109 (+ 10758us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:14.026179 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:14.026491 (+ 312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:14.028403 (+ 1912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:14.028411 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:14.028453 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:34:14.028913 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:14.028928 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:14.030902 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:14.030907 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:14.032988 (+ 2081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:14.033002 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:14.035070 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:14.035079 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:14.035290 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:14.035355 (+ 65us) client_negotiation.cc:769] Sending connection context
0929 20:34:14.035538 (+ 183us) client_negotiation.cc:240] Negotiation successful
0929 20:34:14.035759 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10361,"mutex_wait_us":1,"thread_start_us":11758,"threads_started":1}
I20260929 20:34:14.036317 17634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:14.016020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46822 (local address 127.28.244.129:36123)
0929 20:34:14.026669 (+ 10649us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:14.026679 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:14.026711 (+ 32us) server_negotiation.cc:407] Connection header received
0929 20:34:14.026824 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:14.026838 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:14.027135 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:14.028410 (+ 1275us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:34:14.029148 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:14.030745 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:14.033197 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:14.034798 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:14.035059 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:14.035324 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:14.035656 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 20:34:14.035938 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10453,"thread_start_us":10698,"threads_started":1}
I20260929 20:34:14.037320 17631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:14.015462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.131:33689 (local address 127.0.0.1:52020)
0929 20:34:14.027466 (+ 12004us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:14.027490 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:14.027640 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:14.029070 (+ 1430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:14.029075 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:14.029101 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:34:14.029602 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:14.029615 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:14.031427 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:14.031435 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:14.033751 (+ 2316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:14.033769 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:14.036585 (+ 2816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:14.036591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:14.036852 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:14.036914 (+ 62us) client_negotiation.cc:769] Sending connection context
0929 20:34:14.036981 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 20:34:14.037109 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11814,"thread_start_us":10070,"threads_started":1}
I20260929 20:34:14.038491 17636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:14.016080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52020 (local address 127.28.244.131:33689)
0929 20:34:14.028407 (+ 12327us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:14.028420 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:14.028466 (+ 46us) server_negotiation.cc:407] Connection header received
0929 20:34:14.028602 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:14.028614 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:14.028886 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:14.029081 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:34:14.029864 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:14.031279 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:14.034006 (+ 2727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:14.036356 (+ 2350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:14.037656 (+ 1300us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:14.037849 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:14.037943 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 20:34:14.038123 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12083,"thread_start_us":11892,"threads_started":1}
I20260929 20:34:14.038794 17633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:14.015897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:45457 (local address 127.0.0.1:44518)
0929 20:34:14.029587 (+ 13690us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:14.029607 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:14.030005 (+ 398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:14.030860 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:14.030868 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:14.030895 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:34:14.031471 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:14.031484 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:14.033408 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:14.033414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:14.035649 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:14.035665 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:14.038041 (+ 2376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:14.038049 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:14.038247 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:14.038280 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 20:34:14.038366 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 20:34:14.038506 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13549,"thread_start_us":13434,"threads_started":1}
I20260929 20:34:14.039711 17635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:14.016104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44518 (local address 127.28.244.130:45457)
0929 20:34:14.029088 (+ 12984us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:14.029097 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:14.029728 (+ 631us) server_negotiation.cc:407] Connection header received
0929 20:34:14.030218 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:14.030226 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:14.030455 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:14.030631 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:34:14.031675 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:14.033251 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:14.035957 (+ 2706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:14.037524 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:14.037696 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:14.037905 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:14.039198 (+ 1293us) server_negotiation.cc:299] Negotiation successful
0929 20:34:14.039401 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12807,"thread_start_us":12680,"threads_started":1}
I20260929 20:34:14.041285 17289 tablet_service.cc:1467] Processing CreateTablet for tablet ec1fbfb91524447d9e5a191ae361f2fe (DEFAULT_TABLE table=test-table [id=b7b374c553224442af718c8cfe171a4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:34:14.042323 17552 tablet_service.cc:1467] Processing CreateTablet for tablet ec1fbfb91524447d9e5a191ae361f2fe (DEFAULT_TABLE table=test-table [id=b7b374c553224442af718c8cfe171a4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:34:14.042924 17289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec1fbfb91524447d9e5a191ae361f2fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:34:14.042801 17421 tablet_service.cc:1467] Processing CreateTablet for tablet ec1fbfb91524447d9e5a191ae361f2fe (DEFAULT_TABLE table=test-table [id=b7b374c553224442af718c8cfe171a4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:34:14.043779 17552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec1fbfb91524447d9e5a191ae361f2fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:34:14.044034 17421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec1fbfb91524447d9e5a191ae361f2fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:34:14.056669 17637 tablet_bootstrap.cc:492] T ec1fbfb91524447d9e5a191ae361f2fe P e1dc00bec9be4e04b7bf9818ac33f624: Bootstrap starting.
I20260929 20:34:14.058946 17639 tablet_bootstrap.cc:492] T ec1fbfb91524447d9e5a191ae361f2fe P f127c072fea84fc89b26fb049a560d8a: Bootstrap starting.
I20260929 20:34:14.062327 17637 tablet_bootstrap.cc:654] T ec1fbfb91524447d9e5a191ae361f2fe P e1dc00bec9be4e04b7bf9818ac33f624: Neither blocks nor log segments found. Creating new log.
I20260929 20:34:14.062639 17638 tablet_bootstrap.cc:492] T ec1fbfb91524447d9e5a191ae361f2fe P 194ecaaf435f407fa978f171bc22c86c: Bootstrap starting.
I20260929 20:34:14.062850 17639 tablet_bootstrap.cc:654] T ec1fbfb91524447d9e5a191ae361f2fe P f127c072fea84fc89b26fb049a560d8a: Neither blocks nor log segments found. Creating new log.
I20260929 20:34:14.063884 17637 log.cc:824] T ec1fbfb91524447d9e5a191ae361f2fe P e1dc00bec9be4e04b7bf9818ac33f624: Log is configured to *not* fsync() on all Append() calls
I20260929 20:34:14.064316 17639 log.cc:824] T ec1fbfb91524447d9e5a191ae361f2fe P f127c072fea84fc89b26fb049a560d8a: Log is configured to *not* fsync() on all Append() calls
I20260929 20:34:14.066385 17638 tablet_bootstrap.cc:654] T ec1fbfb91524447d9e5a191ae361f2fe P 194ecaaf435f407fa978f171bc22c86c: Neither blocks nor log segments found. Creating new log.
I20260929 20:34:14.067548 17638 log.cc:824] T ec1fbfb91524447d9e5a191ae361f2fe P 194ecaaf435f407fa978f171bc22c86c: Log is configured to *not* fsync() on all Append() calls
I20260929 20:34:14.068020 17639 tablet_bootstrap.cc:492] T ec1fbfb91524447d9e5a191ae361f2fe P f127c072fea84fc89b26fb049a560d8a: No bootstrap required, opened a new log
I20260929 20:34:14.068020 17637 tablet_bootstrap.cc:492] T ec1fbfb91524447d9e5a191ae361f2fe P e1dc00bec9be4e04b7bf9818ac33f624: No bootstrap required, opened a new log
I20260929 20:34:14.068318 17637 ts_tablet_manager.cc:1397] T ec1fbfb91524447d9e5a191ae361f2fe P e1dc00bec9be4e04b7bf9818ac33f624: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20260929 20:34:14.068318 17639 ts_tablet_manager.cc:1397] T ec1fbfb91524447d9e5a191ae361f2fe P f127c072fea84fc89b26fb049a560d8a: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260929 20:34:14.074651 17638 tablet_bootstrap.cc:492] T ec1fbfb91524447d9e5a191ae361f2fe P 194ecaaf435f407fa978f171bc22c86c: No bootstrap required, opened a new log
I20260929 20:34:14.075073 17638 ts_tablet_manager.cc:1397] T ec1fbfb91524447d9e5a191ae361f2fe P 194ecaaf435f407fa978f171bc22c86c: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.004s
I20260929 20:34:14.079514 17637 raft_consensus.cc:348] T ec1fbfb91524447d9e5a191ae361f2fe P e1dc00bec9be4e04b7bf9818ac33f624 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "194ecaaf435f407fa978f171bc22c86c" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 36123 } } peers { permanent_uuid: "f127c072fea84fc89b26fb049a560d8a" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 33689 } } peers { permanent_uuid: "e1dc00bec9be4e04b7bf9818ac33f624" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45457 } }
I20260929 20:34:14.079514 17639 raft_consensus.cc:348] T ec1fbfb91524447d9e5a191ae361f2fe P f127c072fea84fc89b26fb049a560d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "194ecaaf435f407fa978f171bc22c86c" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 36123 } } peers { permanent_uuid: "f127c072fea84fc89b26fb049a560d8a" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 33689 } } peers { permanent_uuid: "e1dc00bec9be4e04b7bf9818ac33f624" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45457 } }
I20260929 20:34:14.080071 17639 raft_consensus.cc:374] T ec1fbfb91524447d9e5a191ae361f2fe P f127c072fea84fc89b26fb049a560d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:34:14.080072 17637 raft_consensus.cc:374] T ec1fbfb91524447d9e5a191ae361f2fe P e1dc00bec9be4e04b7bf9818ac33f624 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:34:14.080183 17637 raft_consensus.cc:729] T ec1fbfb91524447d9e5a191ae361f2fe P e1dc00bec9be4e04b7bf9818ac33f624 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dc00bec9be4e04b7bf9818ac33f624, State: Initialized, Role: FOLLOWER
I20260929 20:34:14.080240 17639 raft_consensus.cc:729] T ec1fbfb91524447d9e5a191ae361f2fe P f127c072fea84fc89b26fb049a560d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f127c072fea84fc89b26fb049a560d8a, State: Initialized, Role: FOLLOWER
I20260929 20:34:14.080958 17639 consensus_queue.cc:260] T ec1fbfb91524447d9e5a191ae361f2fe P f127c072fea84fc89b26fb049a560d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "194ecaaf435f407fa978f171bc22c86c" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 36123 } } peers { permanent_uuid: "f127c072fea84fc89b26fb049a560d8a" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 33689 } } peers { permanent_uuid: "e1dc00bec9be4e04b7bf9818ac33f624" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45457 } }
I20260929 20:34:14.080919 17637 consensus_queue.cc:260] T ec1fbfb91524447d9e5a191ae361f2fe P e1dc00bec9be4e04b7bf9818ac33f624 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "194ecaaf435f407fa978f171bc22c86c" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 36123 } } peers { permanent_uuid: "f127c072fea84fc89b26fb049a560d8a" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 33689 } } peers { permanent_uuid: "e1dc00bec9be4e04b7bf9818ac33f624" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45457 } }
I20260929 20:34:14.082242 17487 heartbeater.cc:503] Master 127.28.244.190:42259 was elected leader, sending a full tablet report...
I20260929 20:34:14.083268 17637 ts_tablet_manager.cc:1428] T ec1fbfb91524447d9e5a191ae361f2fe P e1dc00bec9be4e04b7bf9818ac33f624: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260929 20:34:14.084326 17619 heartbeater.cc:503] Master 127.28.244.190:42259 was elected leader, sending a full tablet report...
I20260929 20:34:14.086159 17638 raft_consensus.cc:348] T ec1fbfb91524447d9e5a191ae361f2fe P 194ecaaf435f407fa978f171bc22c86c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "194ecaaf435f407fa978f171bc22c86c" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 36123 } } peers { permanent_uuid: "f127c072fea84fc89b26fb049a560d8a" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 33689 } } peers { permanent_uuid: "e1dc00bec9be4e04b7bf9818ac33f624" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45457 } }
I20260929 20:34:14.086632 17639 ts_tablet_manager.cc:1428] T ec1fbfb91524447d9e5a191ae361f2fe P f127c072fea84fc89b26fb049a560d8a: Time spent starting tablet: real 0.018s user 0.013s sys 0.003s
I20260929 20:34:14.086766 17638 raft_consensus.cc:374] T ec1fbfb91524447d9e5a191ae361f2fe P 194ecaaf435f407fa978f171bc22c86c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:34:14.087690 17638 raft_consensus.cc:729] T ec1fbfb91524447d9e5a191ae361f2fe P 194ecaaf435f407fa978f171bc22c86c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 194ecaaf435f407fa978f171bc22c86c, State: Initialized, Role: FOLLOWER
I20260929 20:34:14.088418 17638 consensus_queue.cc:260] T ec1fbfb91524447d9e5a191ae361f2fe P 194ecaaf435f407fa978f171bc22c86c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "194ecaaf435f407fa978f171bc22c86c" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 36123 } } peers { permanent_uuid: "f127c072fea84fc89b26fb049a560d8a" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 33689 } } peers { permanent_uuid: "e1dc00bec9be4e04b7bf9818ac33f624" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45457 } }
I20260929 20:34:14.090456 17638 ts_tablet_manager.cc:1428] T ec1fbfb91524447d9e5a191ae361f2fe P 194ecaaf435f407fa978f171bc22c86c: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
I20260929 20:34:14.115692 17644 raft_consensus.cc:482] T ec1fbfb91524447d9e5a191ae361f2fe P f127c072fea84fc89b26fb049a560d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:34:14.116091 17644 raft_consensus.cc:504] T ec1fbfb91524447d9e5a191ae361f2fe P f127c072fea84fc89b26fb049a560d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "194ecaaf435f407fa978f171bc22c86c" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 36123 } } peers { permanent_uuid: "f127c072fea84fc89b26fb049a560d8a" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 33689 } } peers { permanent_uuid: "e1dc00bec9be4e04b7bf9818ac33f624" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45457 } }
I20260929 20:34:14.117903 17644 leader_election.cc:288] T ec1fbfb91524447d9e5a191ae361f2fe P f127c072fea84fc89b26fb049a560d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 194ecaaf435f407fa978f171bc22c86c (127.28.244.129:36123), e1dc00bec9be4e04b7bf9818ac33f624 (127.28.244.130:45457)
I20260929 20:34:14.123881 17634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:14.118290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:45991 (local address 127.28.244.129:36123)
0929 20:34:14.118553 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:14.118560 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:14.118826 (+ 266us) server_negotiation.cc:407] Connection header received
0929 20:34:14.119066 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:14.119073 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:14.119151 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:14.119253 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:34:14.119657 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:14.120428 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:14.122123 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:14.122949 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:14.123133 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:14.123209 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:14.123462 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 20:34:14.123595 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260929 20:34:14.123925 17621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:14.118290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:36123 (local address 127.28.244.131:45991)
0929 20:34:14.118681 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:14.118706 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:14.118903 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:14.119275 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:14.119280 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:14.119304 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:34:14.119506 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:14.119515 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:14.120563 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:14.120570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:14.121955 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:14.121967 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:14.123120 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:14.123126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:14.123294 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:14.123317 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 20:34:14.123426 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 20:34:14.123573 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270}
I20260929 20:34:14.124477 17635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:14.118479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.244.131:45901 (local address 127.28.244.130:45457)
0929 20:34:14.118683 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:14.118690 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:14.119364 (+ 674us) server_negotiation.cc:407] Connection header received
0929 20:34:14.119598 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:14.119602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:14.119672 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:14.119772 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:34:14.120330 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:14.121042 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:14.122740 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:14.123485 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:14.123649 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:14.123719 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:14.124121 (+ 402us) server_negotiation.cc:299] Negotiation successful
0929 20:34:14.124249 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 20:34:14.125129 17646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:14.118516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.130:45457 (local address 127.28.244.131:45901)
0929 20:34:14.119247 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:14.119260 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:14.119437 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:14.119789 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:14.119793 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:14.119840 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:34:14.120171 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:14.120180 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:14.121201 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:14.121207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:14.122555 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:14.122567 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:14.123664 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:14.123671 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:14.123909 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:14.123933 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 20:34:14.124695 (+ 762us) client_negotiation.cc:240] Negotiation successful
0929 20:34:14.124837 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":419,"threads_started":1}
I20260929 20:34:14.125131 17309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1fbfb91524447d9e5a191ae361f2fe" candidate_uuid: "f127c072fea84fc89b26fb049a560d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "194ecaaf435f407fa978f171bc22c86c" is_pre_election: true
I20260929 20:34:14.125800 17309 raft_consensus.cc:2454] T ec1fbfb91524447d9e5a191ae361f2fe P 194ecaaf435f407fa978f171bc22c86c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f127c072fea84fc89b26fb049a560d8a in term 0.
I20260929 20:34:14.126255 17441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1fbfb91524447d9e5a191ae361f2fe" candidate_uuid: "f127c072fea84fc89b26fb049a560d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dc00bec9be4e04b7bf9818ac33f624" is_pre_election: true
I20260929 20:34:14.126588 17505 leader_election.cc:302] T ec1fbfb91524447d9e5a191ae361f2fe P f127c072fea84fc89b26fb049a560d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 194ecaaf435f407fa978f171bc22c86c, f127c072fea84fc89b26fb049a560d8a; no voters:
I20260929 20:34:14.126796 17441 raft_consensus.cc:2454] T ec1fbfb91524447d9e5a191ae361f2fe P e1dc00bec9be4e04b7bf9818ac33f624 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f127c072fea84fc89b26fb049a560d8a in term 0.
I20260929 20:34:14.127056 17644 raft_consensus.cc:2781] T ec1fbfb91524447d9e5a191ae361f2fe P f127c072fea84fc89b26fb049a560d8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:34:14.127216 17644 raft_consensus.cc:482] T ec1fbfb91524447d9e5a191ae361f2fe P f127c072fea84fc89b26fb049a560d8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:34:14.127342 17644 raft_consensus.cc:3037] T ec1fbfb91524447d9e5a191ae361f2fe P f127c072fea84fc89b26fb049a560d8a [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:34:14.129690 17644 raft_consensus.cc:504] T ec1fbfb91524447d9e5a191ae361f2fe P f127c072fea84fc89b26fb049a560d8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "194ecaaf435f407fa978f171bc22c86c" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 36123 } } peers { permanent_uuid: "f127c072fea84fc89b26fb049a560d8a" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 33689 } } peers { permanent_uuid: "e1dc00bec9be4e04b7bf9818ac33f624" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45457 } }
I20260929 20:34:14.130332 17644 leader_election.cc:288] T ec1fbfb91524447d9e5a191ae361f2fe P f127c072fea84fc89b26fb049a560d8a [CANDIDATE]: Term 1 election: Requested vote from peers 194ecaaf435f407fa978f171bc22c86c (127.28.244.129:36123), e1dc00bec9be4e04b7bf9818ac33f624 (127.28.244.130:45457)
I20260929 20:34:14.130846 17309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1fbfb91524447d9e5a191ae361f2fe" candidate_uuid: "f127c072fea84fc89b26fb049a560d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "194ecaaf435f407fa978f171bc22c86c"
I20260929 20:34:14.130901 17441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1fbfb91524447d9e5a191ae361f2fe" candidate_uuid: "f127c072fea84fc89b26fb049a560d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dc00bec9be4e04b7bf9818ac33f624"
I20260929 20:34:14.131098 17441 raft_consensus.cc:3037] T ec1fbfb91524447d9e5a191ae361f2fe P e1dc00bec9be4e04b7bf9818ac33f624 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:34:14.131091 17309 raft_consensus.cc:3037] T ec1fbfb91524447d9e5a191ae361f2fe P 194ecaaf435f407fa978f171bc22c86c [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:34:14.134469 17441 raft_consensus.cc:2454] T ec1fbfb91524447d9e5a191ae361f2fe P e1dc00bec9be4e04b7bf9818ac33f624 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f127c072fea84fc89b26fb049a560d8a in term 1.
I20260929 20:34:14.134464 17309 raft_consensus.cc:2454] T ec1fbfb91524447d9e5a191ae361f2fe P 194ecaaf435f407fa978f171bc22c86c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f127c072fea84fc89b26fb049a560d8a in term 1.
I20260929 20:34:14.134961 17505 leader_election.cc:302] T ec1fbfb91524447d9e5a191ae361f2fe P f127c072fea84fc89b26fb049a560d8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 194ecaaf435f407fa978f171bc22c86c, f127c072fea84fc89b26fb049a560d8a; no voters:
I20260929 20:34:14.135291 17644 raft_consensus.cc:2781] T ec1fbfb91524447d9e5a191ae361f2fe P f127c072fea84fc89b26fb049a560d8a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:34:14.135843 17644 raft_consensus.cc:686] T ec1fbfb91524447d9e5a191ae361f2fe P f127c072fea84fc89b26fb049a560d8a [term 1 LEADER]: Becoming Leader. State: Replica: f127c072fea84fc89b26fb049a560d8a, State: Running, Role: LEADER
I20260929 20:34:14.136435 17644 consensus_queue.cc:237] T ec1fbfb91524447d9e5a191ae361f2fe P f127c072fea84fc89b26fb049a560d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "194ecaaf435f407fa978f171bc22c86c" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 36123 } } peers { permanent_uuid: "f127c072fea84fc89b26fb049a560d8a" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 33689 } } peers { permanent_uuid: "e1dc00bec9be4e04b7bf9818ac33f624" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45457 } }
I20260929 20:34:14.141373 17167 catalog_manager.cc:5495] T ec1fbfb91524447d9e5a191ae361f2fe P f127c072fea84fc89b26fb049a560d8a reported cstate change: term changed from 0 to 1, leader changed from <none> to f127c072fea84fc89b26fb049a560d8a (127.28.244.131). New cstate: current_term: 1 leader_uuid: "f127c072fea84fc89b26fb049a560d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "194ecaaf435f407fa978f171bc22c86c" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 36123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f127c072fea84fc89b26fb049a560d8a" member_type: VOTER last_known_addr { host: "127.28.244.131" port: 33689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1dc00bec9be4e04b7bf9818ac33f624" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45457 } health_report { overall_health: UNKNOWN } } }
I20260929 20:34:14.166019 17636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:14.160237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52030 (local address 127.28.244.131:33689)
0929 20:34:14.160457 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:14.160464 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:14.160541 (+ 77us) server_negotiation.cc:407] Connection header received
0929 20:34:14.160805 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:14.160817 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:14.160931 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:14.161101 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:34:14.161487 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:14.162229 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:14.163737 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:14.164239 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:14.164382 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:14.164914 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:34:14.165441 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:34:14.165604 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:14.165665 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 20:34:14.165748 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 20:34:14.174814 17441 raft_consensus.cc:1261] T ec1fbfb91524447d9e5a191ae361f2fe P e1dc00bec9be4e04b7bf9818ac33f624 [term 1 FOLLOWER]: Refusing update from remote peer f127c072fea84fc89b26fb049a560d8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:34:14.174881 17309 raft_consensus.cc:1261] T ec1fbfb91524447d9e5a191ae361f2fe P 194ecaaf435f407fa978f171bc22c86c [term 1 FOLLOWER]: Refusing update from remote peer f127c072fea84fc89b26fb049a560d8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:34:14.175908 17647 consensus_queue.cc:1035] T ec1fbfb91524447d9e5a191ae361f2fe P f127c072fea84fc89b26fb049a560d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dc00bec9be4e04b7bf9818ac33f624" member_type: VOTER last_known_addr { host: "127.28.244.130" port: 45457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:34:14.176290 17644 consensus_queue.cc:1035] T ec1fbfb91524447d9e5a191ae361f2fe P f127c072fea84fc89b26fb049a560d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "194ecaaf435f407fa978f171bc22c86c" member_type: VOTER last_known_addr { host: "127.28.244.129" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:34:14.213632 17167 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:34:14.213990 17167 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:34:14.218322 17167 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a71a80705b314928b1412254b8dc672e: Sending DeleteTablet for 3 replicas of tablet ec1fbfb91524447d9e5a191ae361f2fe
I20260929 20:34:14.219631 17289 tablet_service.cc:1514] Processing DeleteTablet for tablet ec1fbfb91524447d9e5a191ae361f2fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:34:14 UTC) from {username='slave'} at 127.0.0.1:46822
I20260929 20:34:14.219726 17552 tablet_service.cc:1514] Processing DeleteTablet for tablet ec1fbfb91524447d9e5a191ae361f2fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:34:14 UTC) from {username='slave'} at 127.0.0.1:52020
I20260929 20:34:14.219966 17421 tablet_service.cc:1514] Processing DeleteTablet for tablet ec1fbfb91524447d9e5a191ae361f2fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:34:14 UTC) from {username='slave'} at 127.0.0.1:44518
I20260929 20:34:14.221848 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 17226
I20260929 20:34:14.232517 17662 tablet_replica.cc:330] T ec1fbfb91524447d9e5a191ae361f2fe P e1dc00bec9be4e04b7bf9818ac33f624: stopping tablet replica
I20260929 20:34:14.233629 17662 raft_consensus.cc:2229] T ec1fbfb91524447d9e5a191ae361f2fe P e1dc00bec9be4e04b7bf9818ac33f624 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:34:14.235566 17662 raft_consensus.cc:2258] T ec1fbfb91524447d9e5a191ae361f2fe P e1dc00bec9be4e04b7bf9818ac33f624 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:34:14.238672 17662 ts_tablet_manager.cc:1905] T ec1fbfb91524447d9e5a191ae361f2fe P e1dc00bec9be4e04b7bf9818ac33f624: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:34:14.240144 17661 tablet_replica.cc:330] T ec1fbfb91524447d9e5a191ae361f2fe P f127c072fea84fc89b26fb049a560d8a: stopping tablet replica
I20260929 20:34:14.240334 17660 tablet_replica.cc:330] T ec1fbfb91524447d9e5a191ae361f2fe P 194ecaaf435f407fa978f171bc22c86c: stopping tablet replica
I20260929 20:34:14.240653 17661 raft_consensus.cc:2229] T ec1fbfb91524447d9e5a191ae361f2fe P f127c072fea84fc89b26fb049a560d8a [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:34:14.240824 17660 raft_consensus.cc:2229] T ec1fbfb91524447d9e5a191ae361f2fe P 194ecaaf435f407fa978f171bc22c86c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:34:14.241457 17660 raft_consensus.cc:2258] T ec1fbfb91524447d9e5a191ae361f2fe P 194ecaaf435f407fa978f171bc22c86c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:34:14.242645 17661 raft_consensus.cc:2258] T ec1fbfb91524447d9e5a191ae361f2fe P f127c072fea84fc89b26fb049a560d8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:34:14.243165 17634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:14.223504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46838 (local address 127.28.244.129:36123)
0929 20:34:14.223710 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:14.223717 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:14.232594 (+ 8877us) server_negotiation.cc:407] Connection header received
0929 20:34:14.233021 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:14.233032 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:14.233174 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:14.233251 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:14.234035 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:14.235451 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:14.237683 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:14.238349 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:14.238463 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:14.241303 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:14.241317 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:14.241334 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:34:14.242194 (+ 860us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:14.242261 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:34:14.242291 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:14.242438 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:14.242660 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 20:34:14.242791 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 20:34:14.244642 17349 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:34:14.245122 17661 ts_tablet_manager.cc:1905] T ec1fbfb91524447d9e5a191ae361f2fe P f127c072fea84fc89b26fb049a560d8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:34:14.250196 17662 ts_tablet_manager.cc:1918] T ec1fbfb91524447d9e5a191ae361f2fe P e1dc00bec9be4e04b7bf9818ac33f624: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:34:14.250466 17662 log.cc:1196] T ec1fbfb91524447d9e5a191ae361f2fe P e1dc00bec9be4e04b7bf9818ac33f624: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-1/wal/wals/ec1fbfb91524447d9e5a191ae361f2fe
I20260929 20:34:14.251097 17662 ts_tablet_manager.cc:1939] T ec1fbfb91524447d9e5a191ae361f2fe P e1dc00bec9be4e04b7bf9818ac33f624: Deleting consensus metadata
I20260929 20:34:14.253391 17152 catalog_manager.cc:4841] TS e1dc00bec9be4e04b7bf9818ac33f624 (127.28.244.130:45457): tablet ec1fbfb91524447d9e5a191ae361f2fe (table test-table [id=b7b374c553224442af718c8cfe171a4b]) successfully deleted
I20260929 20:34:14.256644 17661 ts_tablet_manager.cc:1918] T ec1fbfb91524447d9e5a191ae361f2fe P f127c072fea84fc89b26fb049a560d8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:34:14.256923 17661 log.cc:1196] T ec1fbfb91524447d9e5a191ae361f2fe P f127c072fea84fc89b26fb049a560d8a: Deleting WAL directory at /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713721201454-29650-0/minicluster-data/ts-2/wal/wals/ec1fbfb91524447d9e5a191ae361f2fe
I20260929 20:34:14.257910 17661 ts_tablet_manager.cc:1939] T ec1fbfb91524447d9e5a191ae361f2fe P f127c072fea84fc89b26fb049a560d8a: Deleting consensus metadata
I20260929 20:34:14.259928 17153 catalog_manager.cc:4841] TS f127c072fea84fc89b26fb049a560d8a (127.28.244.131:33689): tablet ec1fbfb91524447d9e5a191ae361f2fe (table test-table [id=b7b374c553224442af718c8cfe171a4b]) successfully deleted
I20260929 20:34:14.244959 17660 ts_tablet_manager.cc:1905] T ec1fbfb91524447d9e5a191ae361f2fe P 194ecaaf435f407fa978f171bc22c86c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:34:14.350824 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 17226
I20260929 20:34:14.371632 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 17358
W20260929 20:34:14.371748 17151 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.244.129:36123 (error 108)
I20260929 20:34:14.374253 17633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:14.373610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:36123 (local address 127.0.0.1:46846)
0929 20:34:14.373903 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:14.374075 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:36123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
W20260929 20:34:14.374562 17151 catalog_manager.cc:4584] TS 194ecaaf435f407fa978f171bc22c86c (127.28.244.129:36123): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ec1fbfb91524447d9e5a191ae361f2fe: Network error: Client connection negotiation failed: client connection to 127.28.244.129:36123: connect: Connection refused (error 111)
I20260929 20:34:14.380756 17635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:14.372652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44520 (local address 127.28.244.130:45457)
0929 20:34:14.372874 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:14.372882 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:14.373226 (+ 344us) server_negotiation.cc:407] Connection header received
0929 20:34:14.373314 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:14.373320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:14.373460 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:14.373612 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:14.374091 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:14.375321 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:14.377017 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:14.377697 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:14.377843 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:14.378988 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:14.379000 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:14.379020 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:34:14.379860 (+ 840us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:14.379934 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:34:14.379961 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:14.380071 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:14.380240 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 20:34:14.380370 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 20:34:14.381592 17482 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:34:14.428414 17633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:14.427747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:36123 (local address 127.0.0.1:46854)
0929 20:34:14.428076 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:14.428189 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:36123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260929 20:34:14.429831 17633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:14.429330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:36123 (local address 127.0.0.1:46860)
0929 20:34:14.429552 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:14.429649 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:36123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260929 20:34:14.490898 17633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:14.490231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:36123 (local address 127.0.0.1:46876)
0929 20:34:14.490517 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:14.490631 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:36123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260929 20:34:14.492120 17633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:14.491684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:36123 (local address 127.0.0.1:46880)
0929 20:34:14.491890 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:14.491985 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:36123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 20:34:14.498478 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 17358
I20260929 20:34:14.517330 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 17490
I20260929 20:34:14.526335 17636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:14.518259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52032 (local address 127.28.244.131:33689)
0929 20:34:14.518463 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:14.518469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:14.518493 (+ 24us) server_negotiation.cc:407] Connection header received
0929 20:34:14.518610 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:14.518615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:14.518729 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:14.518885 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:14.519296 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:14.520640 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:14.522079 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:14.522810 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:14.522955 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:14.524402 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:14.524415 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:14.524441 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:34:14.525366 (+ 925us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:14.525439 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:34:14.525465 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:14.525594 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:14.525791 (+ 197us) server_negotiation.cc:299] Negotiation successful
0929 20:34:14.525904 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 20:34:14.527287 17613 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:34:14.572983 17633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:14.572406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:36123 (local address 127.0.0.1:46884)
0929 20:34:14.572698 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:14.572828 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:36123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260929 20:34:14.574110 17633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:14.573668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.129:36123 (local address 127.0.0.1:46892)
0929 20:34:14.573891 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:14.573978 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.129:36123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260929 20:34:14.653461 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 17490
I20260929 20:34:14.676256 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 17136
I20260929 20:34:14.677839 17197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:34:14.792882 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 17136
2026-09-29T20:34:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2959 ms)
[----------] 6 tests from EncryptionPolicyTest (22832 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:34:14 dist-test-slave-p29l krb5kdc[17671](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:34:14 dist-test-slave-p29l krb5kdc[17671](info): set up 2 sockets
Sep 29 20:34:14 dist-test-slave-p29l krb5kdc[17671](info): commencing operation
krb5kdc: starting...
W20260929 20:34:15.947790 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:34:16 dist-test-slave-p29l krb5kdc[17671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:34:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:34:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:34:16.232429 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39797
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:34:16.586028 17684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:34:16.586344 17684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:34:16.586517 17684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:34:16.594293 17684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:34:16.594422 17684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:34:16.594527 17684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:34:16.594632 17684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:34:16.594693 17684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:34:16.605105 17684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39797
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:34:16.606550 17684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:34:16.607975 17684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:34:16.620719 17690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:16.620671 17691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:16.620769 17693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:34:16.621819 17684 server_base.cc:1034] running on GCE node
I20260929 20:34:16.622690 17684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:34:16.624383 17684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:34:16.625645 17684 hybrid_clock.cc:648] HybridClock initialized: now 1790714056625596 us; error 44 us; skew 500 ppm
I20260929 20:34:16.626065 17684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:34:16 dist-test-slave-p29l krb5kdc[17671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:34:16.629642 17684 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:34:16.631335 17684 webserver.cc:460] Webserver started at http://127.0.0.1:38805/ using document root <none> and password file <none>
I20260929 20:34:16.632174 17684 fs_manager.cc:359] Metadata directory not provided
I20260929 20:34:16.632328 17684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:34:16.632660 17684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:34:16.635973 17684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "370e984fb9804a0291557c88aab0b6a9"
format_stamp: "Formatted at 2026-09-29 20:34:16 on dist-test-slave-p29l"
server_key: "d37488b80fa4a68b697c231c26cd0756"
server_key_iv: "431be06e6357e8392fe5a014056b6088"
server_key_version: "encryptionkey@0"
I20260929 20:34:16.636829 17684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "370e984fb9804a0291557c88aab0b6a9"
format_stamp: "Formatted at 2026-09-29 20:34:16 on dist-test-slave-p29l"
server_key: "d37488b80fa4a68b697c231c26cd0756"
server_key_iv: "431be06e6357e8392fe5a014056b6088"
server_key_version: "encryptionkey@0"
I20260929 20:34:16.642594 17684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 20:34:16.646410 17700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:16.647923 17684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:34:16.648154 17684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "370e984fb9804a0291557c88aab0b6a9"
format_stamp: "Formatted at 2026-09-29 20:34:16 on dist-test-slave-p29l"
server_key: "d37488b80fa4a68b697c231c26cd0756"
server_key_iv: "431be06e6357e8392fe5a014056b6088"
server_key_version: "encryptionkey@0"
I20260929 20:34:16.648409 17684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:34:16.680764 17684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:34:16.685787 17684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:34:16.686267 17684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:34:16.707940 17684 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39797
I20260929 20:34:16.708047 17751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39797 every 8 connection(s)
I20260929 20:34:16.709465 17684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:34:16.709829 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 17684
I20260929 20:34:16.710053 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:34:16.710902 29650 external_mini_cluster.cc:1428] Setting key f95ea292258e8ca1435609360ce72d7c
I20260929 20:34:16.714227 17752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 20:34:16 dist-test-slave-p29l krb5kdc[17671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 20:34:16.728734 17752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 370e984fb9804a0291557c88aab0b6a9: Bootstrap starting.
I20260929 20:34:16.732362 17752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 370e984fb9804a0291557c88aab0b6a9: Neither blocks nor log segments found. Creating new log.
I20260929 20:34:16.733523 17752 log.cc:824] T 00000000000000000000000000000000 P 370e984fb9804a0291557c88aab0b6a9: Log is configured to *not* fsync() on all Append() calls
I20260929 20:34:16.736383 17755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:16.713094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51481 (local address 127.0.0.1:39797)
0929 20:34:16.713940 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:16.713953 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:16.713995 (+ 42us) server_negotiation.cc:407] Connection header received
0929 20:34:16.715726 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:16.715750 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:16.716232 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:16.716566 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:16.718598 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:16.720187 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:16.722147 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:16.722909 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:16.723052 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:16.728071 (+ 5019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:16.728081 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:16.728099 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:16.728143 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:16.731333 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:16.732332 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:16.732339 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:16.732349 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:16.732590 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:16.733120 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:16.733127 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:16.733135 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:16.733742 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:16.734071 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:16.734605 (+ 534us) server_negotiation.cc:299] Negotiation successful
0929 20:34:16.735000 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":407,"threads_started":1}
I20260929 20:34:16.737164 17752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 370e984fb9804a0291557c88aab0b6a9: No bootstrap required, opened a new log
I20260929 20:34:16.740787 29650 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:34:16.745393 17752 raft_consensus.cc:348] T 00000000000000000000000000000000 P 370e984fb9804a0291557c88aab0b6a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "370e984fb9804a0291557c88aab0b6a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39797 } }
I20260929 20:34:16.745982 17752 raft_consensus.cc:374] T 00000000000000000000000000000000 P 370e984fb9804a0291557c88aab0b6a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:34:16.746109 17752 raft_consensus.cc:729] T 00000000000000000000000000000000 P 370e984fb9804a0291557c88aab0b6a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 370e984fb9804a0291557c88aab0b6a9, State: Initialized, Role: FOLLOWER
I20260929 20:34:16.746899 17752 consensus_queue.cc:260] T 00000000000000000000000000000000 P 370e984fb9804a0291557c88aab0b6a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "370e984fb9804a0291557c88aab0b6a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39797 } }
I20260929 20:34:16.747195 17752 raft_consensus.cc:388] T 00000000000000000000000000000000 P 370e984fb9804a0291557c88aab0b6a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:34:16.747354 17752 raft_consensus.cc:482] T 00000000000000000000000000000000 P 370e984fb9804a0291557c88aab0b6a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:34:16.747537 17752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 370e984fb9804a0291557c88aab0b6a9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:34:16.750584 17752 raft_consensus.cc:504] T 00000000000000000000000000000000 P 370e984fb9804a0291557c88aab0b6a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "370e984fb9804a0291557c88aab0b6a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39797 } }
I20260929 20:34:16.751240 17752 leader_election.cc:302] T 00000000000000000000000000000000 P 370e984fb9804a0291557c88aab0b6a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 370e984fb9804a0291557c88aab0b6a9; no voters:
I20260929 20:34:16.752146 17752 leader_election.cc:288] T 00000000000000000000000000000000 P 370e984fb9804a0291557c88aab0b6a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:34:16.752281 17762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 370e984fb9804a0291557c88aab0b6a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:34:16.753234 17762 raft_consensus.cc:686] T 00000000000000000000000000000000 P 370e984fb9804a0291557c88aab0b6a9 [term 1 LEADER]: Becoming Leader. State: Replica: 370e984fb9804a0291557c88aab0b6a9, State: Running, Role: LEADER
I20260929 20:34:16.753832 17762 consensus_queue.cc:237] T 00000000000000000000000000000000 P 370e984fb9804a0291557c88aab0b6a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "370e984fb9804a0291557c88aab0b6a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39797 } }
I20260929 20:34:16.754721 17752 sys_catalog.cc:564] T 00000000000000000000000000000000 P 370e984fb9804a0291557c88aab0b6a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:34:16.757018 17765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 370e984fb9804a0291557c88aab0b6a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 370e984fb9804a0291557c88aab0b6a9. Latest consensus state: current_term: 1 leader_uuid: "370e984fb9804a0291557c88aab0b6a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "370e984fb9804a0291557c88aab0b6a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39797 } } }
I20260929 20:34:16.758471 17765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 370e984fb9804a0291557c88aab0b6a9 [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:16.758059 17764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 370e984fb9804a0291557c88aab0b6a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "370e984fb9804a0291557c88aab0b6a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "370e984fb9804a0291557c88aab0b6a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39797 } } }
I20260929 20:34:16.759202 17764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 370e984fb9804a0291557c88aab0b6a9 [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:16.766563 17755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:16.745783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50649 (local address 127.0.0.1:39797)
0929 20:34:16.745990 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:16.745998 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:16.746353 (+ 355us) server_negotiation.cc:407] Connection header received
0929 20:34:16.746564 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:16.746570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:16.746650 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:16.746761 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:16.748480 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:16.749678 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:16.751049 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:16.751724 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:16.752009 (+ 285us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:16.755397 (+ 3388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:16.755403 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:16.755408 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:16.755451 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:16.760679 (+ 5228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:16.762218 (+ 1539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:16.762226 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:16.762233 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:16.762533 (+ 300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:16.763306 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:16.763314 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:16.763320 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:16.763849 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:16.765507 (+ 1658us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:16.765743 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 20:34:16.766002 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 20:34:16.768620 17770 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:34:16.775345 17770 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:34:16.785939 17770 catalog_manager.cc:1347] Generated new cluster ID: c9868e8c87d6489e8379b4b79e3ea964
I20260929 20:34:16.786049 17770 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:34:16.800774 17770 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:34:16.802220 17770 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:34:16.817453 17770 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 370e984fb9804a0291557c88aab0b6a9: Generated new TSK 0
I20260929 20:34:16.818437 17770 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:34:16.832942 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 17684
I20260929 20:34:16.834498 17747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:34:16.930766 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 17684
W20260929 20:34:16.949580 17669 connection.cc:535] client connection to 127.0.0.1:39797 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39797): Connection reset by peer (error 104)
2026-09-29T20:34:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2117 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:34:17 dist-test-slave-p29l krb5kdc[17785](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:34:17 dist-test-slave-p29l krb5kdc[17785](info): set up 2 sockets
Sep 29 20:34:17 dist-test-slave-p29l krb5kdc[17785](info): commencing operation
krb5kdc: starting...
W20260929 20:34:18.052901 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:34:18 dist-test-slave-p29l krb5kdc[17785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:34:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:34:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:34:18.339695 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40387
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:34:18.701711 17799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:34:18.702059 17799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:34:18.702342 17799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:34:18.710368 17799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:34:18.710520 17799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:34:18.710621 17799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:34:18.710733 17799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:34:18.710799 17799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:34:18.721541 17799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40387
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:34:18.723004 17799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:34:18.724416 17799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:34:18.737552 17804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:18.737672 17805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:18.737586 17807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:34:18.738420 17799 server_base.cc:1034] running on GCE node
I20260929 20:34:18.739336 17799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:34:18.741113 17799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:34:18.742429 17799 hybrid_clock.cc:648] HybridClock initialized: now 1790714058742383 us; error 69 us; skew 500 ppm
I20260929 20:34:18.742893 17799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:34:18 dist-test-slave-p29l krb5kdc[17785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714058, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:34:18.752230 17799 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:34:18.753973 17799 webserver.cc:460] Webserver started at http://127.0.0.1:40493/ using document root <none> and password file <none>
I20260929 20:34:18.754745 17799 fs_manager.cc:359] Metadata directory not provided
I20260929 20:34:18.754912 17799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:34:18.755239 17799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:34:18.758653 17799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "5907712e5fc24fbeb8956dd825fb6991"
format_stamp: "Formatted at 2026-09-29 20:34:18 on dist-test-slave-p29l"
server_key: "a111266ac293fc97ebafdf71ab206883"
server_key_iv: "da27c7fcf090c1b3fd346ff211dfd55c"
server_key_version: "encryptionkey@0"
I20260929 20:34:18.759521 17799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "5907712e5fc24fbeb8956dd825fb6991"
format_stamp: "Formatted at 2026-09-29 20:34:18 on dist-test-slave-p29l"
server_key: "a111266ac293fc97ebafdf71ab206883"
server_key_iv: "da27c7fcf090c1b3fd346ff211dfd55c"
server_key_version: "encryptionkey@0"
I20260929 20:34:18.765394 17799 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260929 20:34:18.769358 17814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:18.770946 17799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:34:18.771148 17799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "5907712e5fc24fbeb8956dd825fb6991"
format_stamp: "Formatted at 2026-09-29 20:34:18 on dist-test-slave-p29l"
server_key: "a111266ac293fc97ebafdf71ab206883"
server_key_iv: "da27c7fcf090c1b3fd346ff211dfd55c"
server_key_version: "encryptionkey@0"
I20260929 20:34:18.771377 17799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:34:18.786379 17799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:34:18.791152 17799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:34:18.791464 17799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:34:18.811390 17799 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40387
I20260929 20:34:18.811430 17865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40387 every 8 connection(s)
I20260929 20:34:18.812947 17799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:34:18.817154 17866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:34:18.818217 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 17799
I20260929 20:34:18.818419 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:34:18.818918 29650 external_mini_cluster.cc:1428] Setting key 8b3b0c40e8b9d6bdc185f55b810a42a9
I20260929 20:34:18.832705 17866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5907712e5fc24fbeb8956dd825fb6991: Bootstrap starting.
I20260929 20:34:18.834450 17869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:18.821138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39993 (local address 127.0.0.1:40387)
0929 20:34:18.822062 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:18.822075 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:18.822122 (+ 47us) server_negotiation.cc:407] Connection header received
0929 20:34:18.824062 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:18.824091 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:18.824612 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:18.825008 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:18.825820 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:18.827243 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:18.829382 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:18.830201 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:18.830364 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:18.831953 (+ 1589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:18.831963 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:18.831981 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:34:18.832571 (+ 590us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:18.832664 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:34:18.832695 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:18.832887 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:18.833158 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 20:34:18.833396 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":416,"threads_started":1}
I20260929 20:34:18.836320 17866 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5907712e5fc24fbeb8956dd825fb6991: Neither blocks nor log segments found. Creating new log.
I20260929 20:34:18.837598 17866 log.cc:824] T 00000000000000000000000000000000 P 5907712e5fc24fbeb8956dd825fb6991: Log is configured to *not* fsync() on all Append() calls
I20260929 20:34:18.838837 29650 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:34:18.840826 17866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5907712e5fc24fbeb8956dd825fb6991: No bootstrap required, opened a new log
I20260929 20:34:18.852129 17869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:18.843955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38497 (local address 127.0.0.1:40387)
0929 20:34:18.844250 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:18.844260 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:18.844667 (+ 407us) server_negotiation.cc:407] Connection header received
0929 20:34:18.844966 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:18.844972 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:18.845061 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:18.845180 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:18.845824 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:18.847260 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:18.848741 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:18.849579 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:18.849729 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:18.851111 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:18.851118 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:18.851125 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:34:18.851150 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:18.851245 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:34:18.851269 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:18.851440 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:18.851655 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 20:34:18.851718 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:34:18.852407 17866 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5907712e5fc24fbeb8956dd825fb6991 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5907712e5fc24fbeb8956dd825fb6991" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40387 } }
I20260929 20:34:18.853008 17866 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5907712e5fc24fbeb8956dd825fb6991 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:34:18.853158 17866 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5907712e5fc24fbeb8956dd825fb6991 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5907712e5fc24fbeb8956dd825fb6991, State: Initialized, Role: FOLLOWER
I20260929 20:34:18.853951 17866 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5907712e5fc24fbeb8956dd825fb6991 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5907712e5fc24fbeb8956dd825fb6991" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40387 } }
I20260929 20:34:18.854230 17866 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5907712e5fc24fbeb8956dd825fb6991 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:34:18.854414 17866 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5907712e5fc24fbeb8956dd825fb6991 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:34:18.854624 17866 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5907712e5fc24fbeb8956dd825fb6991 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:34:18.857270 17866 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5907712e5fc24fbeb8956dd825fb6991 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5907712e5fc24fbeb8956dd825fb6991" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40387 } }
I20260929 20:34:18.858012 17866 leader_election.cc:302] T 00000000000000000000000000000000 P 5907712e5fc24fbeb8956dd825fb6991 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5907712e5fc24fbeb8956dd825fb6991; no voters:
I20260929 20:34:18.858829 17866 leader_election.cc:288] T 00000000000000000000000000000000 P 5907712e5fc24fbeb8956dd825fb6991 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:34:18.859066 17876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5907712e5fc24fbeb8956dd825fb6991 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:34:18.859944 17876 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5907712e5fc24fbeb8956dd825fb6991 [term 1 LEADER]: Becoming Leader. State: Replica: 5907712e5fc24fbeb8956dd825fb6991, State: Running, Role: LEADER
I20260929 20:34:18.860539 17876 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5907712e5fc24fbeb8956dd825fb6991 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5907712e5fc24fbeb8956dd825fb6991" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40387 } }
I20260929 20:34:18.861384 17866 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5907712e5fc24fbeb8956dd825fb6991 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:34:18.863407 17878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5907712e5fc24fbeb8956dd825fb6991 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5907712e5fc24fbeb8956dd825fb6991. Latest consensus state: current_term: 1 leader_uuid: "5907712e5fc24fbeb8956dd825fb6991" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5907712e5fc24fbeb8956dd825fb6991" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40387 } } }
I20260929 20:34:18.863837 17878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5907712e5fc24fbeb8956dd825fb6991 [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:18.864337 17877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5907712e5fc24fbeb8956dd825fb6991 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5907712e5fc24fbeb8956dd825fb6991" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5907712e5fc24fbeb8956dd825fb6991" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40387 } } }
I20260929 20:34:18.864650 17877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5907712e5fc24fbeb8956dd825fb6991 [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:18.868206 17881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:34:18.873911 17881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:34:18.883304 17881 catalog_manager.cc:1347] Generated new cluster ID: f977e4628a61467fbf43b7d2200322b3
I20260929 20:34:18.883448 17881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:34:18.902815 17881 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:34:18.904352 17881 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:34:18.911581 17881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5907712e5fc24fbeb8956dd825fb6991: Generated new TSK 0
I20260929 20:34:18.912604 17881 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:34:18.920845 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 17799
W20260929 20:34:18.922187 17861 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39993
W20260929 20:34:18.923012 29650 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 17799: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:34:18.923206 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 17799
W20260929 20:34:18.941665 17783 connection.cc:535] client connection to 127.0.0.1:40387 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40387): Connection reset by peer (error 104)
2026-09-29T20:34:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1992 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:34:19 dist-test-slave-p29l krb5kdc[17897](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:34:19 dist-test-slave-p29l krb5kdc[17897](info): set up 2 sockets
Sep 29 20:34:19 dist-test-slave-p29l krb5kdc[17897](info): commencing operation
krb5kdc: starting...
W20260929 20:34:20.048717 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:34:20 dist-test-slave-p29l krb5kdc[17897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:34:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:34:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:34:20.337185 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34305
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:34:20.691511 17911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:34:20.691928 17911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:34:20.692147 17911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:34:20.701064 17911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:34:20.701200 17911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:34:20.701246 17911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:34:20.701304 17911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:34:20.701349 17911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:34:20.711706 17911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34305
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:34:20.713191 17911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:34:20.714557 17911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:34:20.727257 17916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:20.727366 17919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:20.727391 17917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:34:20.728389 17911 server_base.cc:1034] running on GCE node
I20260929 20:34:20.729302 17911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:34:20.730942 17911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:34:20.732189 17911 hybrid_clock.cc:648] HybridClock initialized: now 1790714060732134 us; error 49 us; skew 500 ppm
I20260929 20:34:20.732584 17911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:34:20 dist-test-slave-p29l krb5kdc[17897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714060, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:34:20.736239 17911 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:34:20.738354 17911 webserver.cc:460] Webserver started at http://127.0.0.1:40423/ using document root <none> and password file <none>
I20260929 20:34:20.739223 17911 fs_manager.cc:359] Metadata directory not provided
I20260929 20:34:20.739408 17911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:34:20.739792 17911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:34:20.743414 17911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "7130c3fb699a42889b244e2ea85370e4"
format_stamp: "Formatted at 2026-09-29 20:34:20 on dist-test-slave-p29l"
server_key: "e7bea12d8784b16617d2b6205a5e4a92"
server_key_iv: "3bd419a9c0227fb2c9fd907ab2742aba"
server_key_version: "encryptionkey@0"
I20260929 20:34:20.744450 17911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "7130c3fb699a42889b244e2ea85370e4"
format_stamp: "Formatted at 2026-09-29 20:34:20 on dist-test-slave-p29l"
server_key: "e7bea12d8784b16617d2b6205a5e4a92"
server_key_iv: "3bd419a9c0227fb2c9fd907ab2742aba"
server_key_version: "encryptionkey@0"
I20260929 20:34:20.750563 17911 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260929 20:34:20.754796 17926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:20.756385 17911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 20:34:20.756623 17911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "7130c3fb699a42889b244e2ea85370e4"
format_stamp: "Formatted at 2026-09-29 20:34:20 on dist-test-slave-p29l"
server_key: "e7bea12d8784b16617d2b6205a5e4a92"
server_key_iv: "3bd419a9c0227fb2c9fd907ab2742aba"
server_key_version: "encryptionkey@0"
I20260929 20:34:20.756858 17911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:34:20.763216 17911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:34:20.767987 17911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:34:20.768322 17911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:34:20.788794 17911 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34305
I20260929 20:34:20.788877 17977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34305 every 8 connection(s)
I20260929 20:34:20.790401 17911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:34:20.794791 17978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:34:20.795327 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 17911
I20260929 20:34:20.795543 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:34:20.796200 29650 external_mini_cluster.cc:1428] Setting key cd948b07adae9b4c3df89c0a707460b8
I20260929 20:34:20.805511 17981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:20.798164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52625 (local address 127.0.0.1:34305)
0929 20:34:20.799084 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:20.799096 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:20.799140 (+ 44us) server_negotiation.cc:407] Connection header received
0929 20:34:20.801794 (+ 2654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:20.801825 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:20.802221 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:20.802607 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:20.803795 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:20.803801 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:20.803907 (+ 106us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:34:20.803925 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:20.803991 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:34:20.804014 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:20.804187 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:20.804323 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 20:34:20.804529 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":456,"threads_started":1}
I20260929 20:34:20.809559 29650 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:34:20.810141 17978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7130c3fb699a42889b244e2ea85370e4: Bootstrap starting.
I20260929 20:34:20.815017 17978 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7130c3fb699a42889b244e2ea85370e4: Neither blocks nor log segments found. Creating new log.
I20260929 20:34:20.816522 17978 log.cc:824] T 00000000000000000000000000000000 P 7130c3fb699a42889b244e2ea85370e4: Log is configured to *not* fsync() on all Append() calls
I20260929 20:34:20.819602 17981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:20.815507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32783 (local address 127.0.0.1:34305)
0929 20:34:20.815742 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:20.815750 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:20.817425 (+ 1675us) server_negotiation.cc:407] Connection header received
0929 20:34:20.817611 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:20.817619 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:20.817691 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:20.817834 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:20.818887 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:20.818893 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:20.818898 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:34:20.818915 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:20.818956 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:34:20.818971 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:20.819069 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:20.819228 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 20:34:20.819280 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 20:34:20.820531 17978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7130c3fb699a42889b244e2ea85370e4: No bootstrap required, opened a new log
I20260929 20:34:20.828958 17978 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7130c3fb699a42889b244e2ea85370e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7130c3fb699a42889b244e2ea85370e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34305 } }
I20260929 20:34:20.829468 17978 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7130c3fb699a42889b244e2ea85370e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:34:20.829591 17978 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7130c3fb699a42889b244e2ea85370e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7130c3fb699a42889b244e2ea85370e4, State: Initialized, Role: FOLLOWER
I20260929 20:34:20.830251 17978 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7130c3fb699a42889b244e2ea85370e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7130c3fb699a42889b244e2ea85370e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34305 } }
I20260929 20:34:20.830543 17978 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7130c3fb699a42889b244e2ea85370e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:34:20.830706 17978 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7130c3fb699a42889b244e2ea85370e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:34:20.830906 17978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7130c3fb699a42889b244e2ea85370e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:34:20.833587 17978 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7130c3fb699a42889b244e2ea85370e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7130c3fb699a42889b244e2ea85370e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34305 } }
I20260929 20:34:20.834172 17978 leader_election.cc:302] T 00000000000000000000000000000000 P 7130c3fb699a42889b244e2ea85370e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7130c3fb699a42889b244e2ea85370e4; no voters:
I20260929 20:34:20.835021 17978 leader_election.cc:288] T 00000000000000000000000000000000 P 7130c3fb699a42889b244e2ea85370e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:34:20.835162 17988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7130c3fb699a42889b244e2ea85370e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:34:20.835986 17988 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7130c3fb699a42889b244e2ea85370e4 [term 1 LEADER]: Becoming Leader. State: Replica: 7130c3fb699a42889b244e2ea85370e4, State: Running, Role: LEADER
I20260929 20:34:20.836719 17988 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7130c3fb699a42889b244e2ea85370e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7130c3fb699a42889b244e2ea85370e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34305 } }
I20260929 20:34:20.837787 17978 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7130c3fb699a42889b244e2ea85370e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:34:20.842542 17989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7130c3fb699a42889b244e2ea85370e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7130c3fb699a42889b244e2ea85370e4. Latest consensus state: current_term: 1 leader_uuid: "7130c3fb699a42889b244e2ea85370e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7130c3fb699a42889b244e2ea85370e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34305 } } }
I20260929 20:34:20.842438 17990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7130c3fb699a42889b244e2ea85370e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7130c3fb699a42889b244e2ea85370e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7130c3fb699a42889b244e2ea85370e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34305 } } }
I20260929 20:34:20.843166 17989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7130c3fb699a42889b244e2ea85370e4 [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:20.843230 17990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7130c3fb699a42889b244e2ea85370e4 [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:20.849819 17995 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:34:20.855307 17995 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:34:20.865640 17995 catalog_manager.cc:1347] Generated new cluster ID: 09647b7db2f849ae918af9b7150631a2
I20260929 20:34:20.865804 17995 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:34:20.894487 17995 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:34:20.895012 17995 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:34:20.907099 17995 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7130c3fb699a42889b244e2ea85370e4: Generated new TSK 0
I20260929 20:34:20.908174 17995 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:34:20.918148 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 17911
W20260929 20:34:20.919348 17973 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52625
W20260929 20:34:20.920383 29650 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 17911: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:34:20.920562 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 17911
2026-09-29T20:34:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1997 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:34:21 dist-test-slave-p29l krb5kdc[18009](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:34:21 dist-test-slave-p29l krb5kdc[18009](info): set up 2 sockets
Sep 29 20:34:21 dist-test-slave-p29l krb5kdc[18009](info): commencing operation
krb5kdc: starting...
W20260929 20:34:22.080075 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.005s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:34:22 dist-test-slave-p29l krb5kdc[18009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:34:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:34:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:34:22.363628 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39407
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:34:22.719897 18022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:34:22.720265 18022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:34:22.720445 18022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:34:22.728101 18022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:34:22.728224 18022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:34:22.728330 18022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:34:22.728436 18022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:34:22.728519 18022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:34:22.738978 18022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:34:22.740479 18022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:34:22.741814 18022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:34:22.754302 18028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:22.754370 18027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:22.754387 18030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:34:22.755283 18022 server_base.cc:1034] running on GCE node
I20260929 20:34:22.756393 18022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:34:22.758165 18022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:34:22.759433 18022 hybrid_clock.cc:648] HybridClock initialized: now 1790714062759379 us; error 43 us; skew 500 ppm
I20260929 20:34:22.759977 18022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:34:22 dist-test-slave-p29l krb5kdc[18009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714062, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:34:22.763538 18022 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:34:22.765372 18022 webserver.cc:460] Webserver started at http://127.0.0.1:43081/ using document root <none> and password file <none>
I20260929 20:34:22.766192 18022 fs_manager.cc:359] Metadata directory not provided
I20260929 20:34:22.766382 18022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:34:22.766746 18022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:34:22.770371 18022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "9e958dddbb9d4c28a42f7d6b5742b64b"
format_stamp: "Formatted at 2026-09-29 20:34:22 on dist-test-slave-p29l"
server_key: "ee380ff1f85717257d5bdf1574b684ad"
server_key_iv: "f40fbcc5e35137080cf4c8293cc353c7"
server_key_version: "encryptionkey@0"
I20260929 20:34:22.771334 18022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "9e958dddbb9d4c28a42f7d6b5742b64b"
format_stamp: "Formatted at 2026-09-29 20:34:22 on dist-test-slave-p29l"
server_key: "ee380ff1f85717257d5bdf1574b684ad"
server_key_iv: "f40fbcc5e35137080cf4c8293cc353c7"
server_key_version: "encryptionkey@0"
I20260929 20:34:22.777633 18022 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260929 20:34:22.781807 18037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:22.783341 18022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:34:22.783568 18022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "9e958dddbb9d4c28a42f7d6b5742b64b"
format_stamp: "Formatted at 2026-09-29 20:34:22 on dist-test-slave-p29l"
server_key: "ee380ff1f85717257d5bdf1574b684ad"
server_key_iv: "f40fbcc5e35137080cf4c8293cc353c7"
server_key_version: "encryptionkey@0"
I20260929 20:34:22.783795 18022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:34:22.806748 18022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:34:22.811756 18022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:34:22.812187 18022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:34:22.834208 18022 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39407
I20260929 20:34:22.834230 18089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39407 every 8 connection(s)
I20260929 20:34:22.835857 18022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:34:22.840041 18090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:34:22.841293 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 18022
I20260929 20:34:22.841490 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:34:22.842063 29650 external_mini_cluster.cc:1428] Setting key c41225dbd27d3d0f5771f53f5e9cae87
I20260929 20:34:22.855706 18090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e958dddbb9d4c28a42f7d6b5742b64b: Bootstrap starting.
Sep 29 20:34:22 dist-test-slave-p29l krb5kdc[18009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 20:34:22.859969 18090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e958dddbb9d4c28a42f7d6b5742b64b: Neither blocks nor log segments found. Creating new log.
I20260929 20:34:22.861547 18090 log.cc:824] T 00000000000000000000000000000000 P 9e958dddbb9d4c28a42f7d6b5742b64b: Log is configured to *not* fsync() on all Append() calls
I20260929 20:34:22.865411 18090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e958dddbb9d4c28a42f7d6b5742b64b: No bootstrap required, opened a new log
I20260929 20:34:22.868602 18093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:22.844174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34439 (local address 127.0.0.1:39407)
0929 20:34:22.845076 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:22.845088 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:22.845135 (+ 47us) server_negotiation.cc:407] Connection header received
0929 20:34:22.846823 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:22.846848 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:22.847312 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:22.847690 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:22.849727 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:22.851278 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:22.853334 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:22.854073 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:22.854245 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:22.859274 (+ 5029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:22.859286 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:22.859307 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:22.859360 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:22.863515 (+ 4155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:22.864490 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:22.864499 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:22.864508 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:22.864730 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:22.865152 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:22.865157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:22.865160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:22.865885 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:22.866246 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:22.866626 (+ 380us) server_negotiation.cc:299] Negotiation successful
0929 20:34:22.867097 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":403,"threads_started":1}
I20260929 20:34:22.872972 29650 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:34:22.875291 18090 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e958dddbb9d4c28a42f7d6b5742b64b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e958dddbb9d4c28a42f7d6b5742b64b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39407 } }
I20260929 20:34:22.875712 18090 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e958dddbb9d4c28a42f7d6b5742b64b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:34:22.875859 18090 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e958dddbb9d4c28a42f7d6b5742b64b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e958dddbb9d4c28a42f7d6b5742b64b, State: Initialized, Role: FOLLOWER
I20260929 20:34:22.876572 18090 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e958dddbb9d4c28a42f7d6b5742b64b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e958dddbb9d4c28a42f7d6b5742b64b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39407 } }
I20260929 20:34:22.876844 18090 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e958dddbb9d4c28a42f7d6b5742b64b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:34:22.877002 18090 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e958dddbb9d4c28a42f7d6b5742b64b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:34:22.877187 18090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e958dddbb9d4c28a42f7d6b5742b64b [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:34:22.879623 18090 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e958dddbb9d4c28a42f7d6b5742b64b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e958dddbb9d4c28a42f7d6b5742b64b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39407 } }
I20260929 20:34:22.880304 18090 leader_election.cc:302] T 00000000000000000000000000000000 P 9e958dddbb9d4c28a42f7d6b5742b64b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e958dddbb9d4c28a42f7d6b5742b64b; no voters:
I20260929 20:34:22.881016 18090 leader_election.cc:288] T 00000000000000000000000000000000 P 9e958dddbb9d4c28a42f7d6b5742b64b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:34:22.881121 18099 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e958dddbb9d4c28a42f7d6b5742b64b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:34:22.883755 18099 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e958dddbb9d4c28a42f7d6b5742b64b [term 1 LEADER]: Becoming Leader. State: Replica: 9e958dddbb9d4c28a42f7d6b5742b64b, State: Running, Role: LEADER
I20260929 20:34:22.884011 18090 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e958dddbb9d4c28a42f7d6b5742b64b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:34:22.884490 18099 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e958dddbb9d4c28a42f7d6b5742b64b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e958dddbb9d4c28a42f7d6b5742b64b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39407 } }
I20260929 20:34:22.887348 18102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e958dddbb9d4c28a42f7d6b5742b64b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e958dddbb9d4c28a42f7d6b5742b64b. Latest consensus state: current_term: 1 leader_uuid: "9e958dddbb9d4c28a42f7d6b5742b64b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e958dddbb9d4c28a42f7d6b5742b64b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39407 } } }
I20260929 20:34:22.887792 18102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e958dddbb9d4c28a42f7d6b5742b64b [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:22.888403 18100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e958dddbb9d4c28a42f7d6b5742b64b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e958dddbb9d4c28a42f7d6b5742b64b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e958dddbb9d4c28a42f7d6b5742b64b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39407 } } }
I20260929 20:34:22.888708 18100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e958dddbb9d4c28a42f7d6b5742b64b [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:22.897738 18107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:34:22.907269 18093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:22.881295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60269 (local address 127.0.0.1:39407)
0929 20:34:22.881515 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:22.881525 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:22.883080 (+ 1555us) server_negotiation.cc:407] Connection header received
0929 20:34:22.883320 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:22.883329 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:22.883435 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:22.883569 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:22.885933 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:22.887544 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:22.890329 (+ 2785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:22.891306 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:22.891452 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:22.897616 (+ 6164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:22.897628 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:22.897636 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:22.897700 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:22.902915 (+ 5215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:22.903958 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:22.903966 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:22.903974 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:22.904139 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:22.904776 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:22.904785 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:22.904790 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:22.905464 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:22.905944 (+ 480us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:22.906614 (+ 670us) server_negotiation.cc:299] Negotiation successful
0929 20:34:22.906881 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:34:22.909585 18107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:34:22.919977 18107 catalog_manager.cc:1347] Generated new cluster ID: bc915efa77b64d23a53654f628a76e1e
I20260929 20:34:22.920122 18107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:34:22.928395 18107 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:34:22.930403 18107 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:34:22.942010 18107 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e958dddbb9d4c28a42f7d6b5742b64b: Generated new TSK 0
I20260929 20:34:22.943019 18107 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:34:22.956719 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 18022
I20260929 20:34:22.958290 18085 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:34:23.050523 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 18022
2026-09-29T20:34:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2130 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:34:23 dist-test-slave-p29l krb5kdc[18123](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:34:23 dist-test-slave-p29l krb5kdc[18123](info): set up 2 sockets
Sep 29 20:34:23 dist-test-slave-p29l krb5kdc[18123](info): commencing operation
krb5kdc: starting...
W20260929 20:34:24.171887 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.002s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:34:24 dist-test-slave-p29l krb5kdc[18123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:34:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:34:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:34:24.451740 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42353
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:34:24.805805 18136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:34:24.806131 18136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:34:24.806304 18136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:34:24.813961 18136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:34:24.814091 18136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:34:24.814208 18136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:34:24.814316 18136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:34:24.814395 18136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:34:24.824600 18136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:34:24.826045 18136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:34:24.827412 18136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:34:24.840960 18142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:24.841212 18141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:24.841452 18144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:34:24.842765 18136 server_base.cc:1034] running on GCE node
I20260929 20:34:24.844029 18136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:34:24.846377 18136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:34:24.847759 18136 hybrid_clock.cc:648] HybridClock initialized: now 1790714064847579 us; error 177 us; skew 500 ppm
I20260929 20:34:24.848502 18136 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:34:24 dist-test-slave-p29l krb5kdc[18123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:34:24.853816 18136 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:34:24.855674 18136 webserver.cc:460] Webserver started at http://127.0.0.1:33315/ using document root <none> and password file <none>
I20260929 20:34:24.856803 18136 fs_manager.cc:359] Metadata directory not provided
I20260929 20:34:24.856956 18136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:34:24.857379 18136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:34:24.862035 18136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "56a9df0c7471453a827cb22563f5cfe2"
format_stamp: "Formatted at 2026-09-29 20:34:24 on dist-test-slave-p29l"
server_key: "ff88b8f13c6c0d1741c6e912e1b17566"
server_key_iv: "76feea1aa21ba9cb3d4c1f60d31e3040"
server_key_version: "encryptionkey@0"
I20260929 20:34:24.863117 18136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "56a9df0c7471453a827cb22563f5cfe2"
format_stamp: "Formatted at 2026-09-29 20:34:24 on dist-test-slave-p29l"
server_key: "ff88b8f13c6c0d1741c6e912e1b17566"
server_key_iv: "76feea1aa21ba9cb3d4c1f60d31e3040"
server_key_version: "encryptionkey@0"
I20260929 20:34:24.871340 18136 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.004s
I20260929 20:34:24.877292 18151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:24.879662 18136 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:34:24.879942 18136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "56a9df0c7471453a827cb22563f5cfe2"
format_stamp: "Formatted at 2026-09-29 20:34:24 on dist-test-slave-p29l"
server_key: "ff88b8f13c6c0d1741c6e912e1b17566"
server_key_iv: "76feea1aa21ba9cb3d4c1f60d31e3040"
server_key_version: "encryptionkey@0"
I20260929 20:34:24.880256 18136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:34:24.914115 18136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:34:24.919039 18136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:34:24.919310 18136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:34:24.938669 18136 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42353
I20260929 20:34:24.938699 18203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42353 every 8 connection(s)
I20260929 20:34:24.940291 18136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:34:24.944555 18204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:34:24.949949 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 18136
I20260929 20:34:24.950107 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:34:24.950596 29650 external_mini_cluster.cc:1428] Setting key d5a292db1646273d6becc338cb9b5f4c
I20260929 20:34:24.956652 18204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56a9df0c7471453a827cb22563f5cfe2: Bootstrap starting.
I20260929 20:34:24.960002 18204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56a9df0c7471453a827cb22563f5cfe2: Neither blocks nor log segments found. Creating new log.
I20260929 20:34:24.961117 18204 log.cc:824] T 00000000000000000000000000000000 P 56a9df0c7471453a827cb22563f5cfe2: Log is configured to *not* fsync() on all Append() calls
I20260929 20:34:24.964901 18207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:24.952631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49905 (local address 127.0.0.1:42353)
0929 20:34:24.953544 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:24.953556 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:24.953598 (+ 42us) server_negotiation.cc:407] Connection header received
0929 20:34:24.955448 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:24.955475 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:24.955985 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:24.956413 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:24.956968 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:24.958311 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:24.959686 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:24.960535 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:24.960730 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:24.962099 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:24.962107 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:24.962122 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:34:24.962770 (+ 648us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:24.962874 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:34:24.962906 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:24.963038 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:24.963263 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 20:34:24.963564 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":401,"threads_started":1}
I20260929 20:34:24.964990 18204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56a9df0c7471453a827cb22563f5cfe2: No bootstrap required, opened a new log
I20260929 20:34:24.968992 29650 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:34:24.973165 18204 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56a9df0c7471453a827cb22563f5cfe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56a9df0c7471453a827cb22563f5cfe2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42353 } }
I20260929 20:34:24.973708 18204 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56a9df0c7471453a827cb22563f5cfe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:34:24.973819 18204 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56a9df0c7471453a827cb22563f5cfe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56a9df0c7471453a827cb22563f5cfe2, State: Initialized, Role: FOLLOWER
I20260929 20:34:24.974587 18204 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56a9df0c7471453a827cb22563f5cfe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56a9df0c7471453a827cb22563f5cfe2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42353 } }
I20260929 20:34:24.974903 18204 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56a9df0c7471453a827cb22563f5cfe2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:34:24.975065 18204 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56a9df0c7471453a827cb22563f5cfe2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:34:24.975268 18204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56a9df0c7471453a827cb22563f5cfe2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:34:24.978609 18204 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56a9df0c7471453a827cb22563f5cfe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56a9df0c7471453a827cb22563f5cfe2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42353 } }
I20260929 20:34:24.979256 18204 leader_election.cc:302] T 00000000000000000000000000000000 P 56a9df0c7471453a827cb22563f5cfe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56a9df0c7471453a827cb22563f5cfe2; no voters:
I20260929 20:34:24.980173 18204 leader_election.cc:288] T 00000000000000000000000000000000 P 56a9df0c7471453a827cb22563f5cfe2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:34:24.980324 18214 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56a9df0c7471453a827cb22563f5cfe2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:34:24.981184 18207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:24.973519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48921 (local address 127.0.0.1:42353)
0929 20:34:24.973747 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:24.973753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:24.974330 (+ 577us) server_negotiation.cc:407] Connection header received
0929 20:34:24.974540 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:24.974545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:24.974613 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:24.974694 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:24.975342 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:24.976232 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:24.978199 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:24.978695 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:24.978863 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:24.980167 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:24.980176 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:24.980182 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:34:24.980210 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:24.980261 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:34:24.980287 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:24.980402 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:24.980601 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 20:34:24.980718 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:34:24.981607 18214 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56a9df0c7471453a827cb22563f5cfe2 [term 1 LEADER]: Becoming Leader. State: Replica: 56a9df0c7471453a827cb22563f5cfe2, State: Running, Role: LEADER
I20260929 20:34:24.982261 18214 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56a9df0c7471453a827cb22563f5cfe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56a9df0c7471453a827cb22563f5cfe2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42353 } }
I20260929 20:34:24.984825 18204 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56a9df0c7471453a827cb22563f5cfe2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:34:24.985253 18216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56a9df0c7471453a827cb22563f5cfe2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56a9df0c7471453a827cb22563f5cfe2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56a9df0c7471453a827cb22563f5cfe2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42353 } } }
I20260929 20:34:24.985613 18216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56a9df0c7471453a827cb22563f5cfe2 [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:24.986348 18215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56a9df0c7471453a827cb22563f5cfe2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56a9df0c7471453a827cb22563f5cfe2. Latest consensus state: current_term: 1 leader_uuid: "56a9df0c7471453a827cb22563f5cfe2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56a9df0c7471453a827cb22563f5cfe2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42353 } } }
I20260929 20:34:24.986649 18215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56a9df0c7471453a827cb22563f5cfe2 [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:24.988730 18218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:34:24.994154 18218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:34:25.014951 18218 catalog_manager.cc:1347] Generated new cluster ID: 5bececd726a4446088b82461d958c49e
I20260929 20:34:25.015101 18218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:34:25.037346 18218 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:34:25.039395 18218 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:34:25.051472 18218 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56a9df0c7471453a827cb22563f5cfe2: Generated new TSK 0
I20260929 20:34:25.052666 18218 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:34:25.059309 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 18136
W20260929 20:34:25.060633 18199 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49905
W20260929 20:34:25.061511 29650 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 18136: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:34:25.061677 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 18136
2026-09-29T20:34:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2011 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:34:25 dist-test-slave-p29l krb5kdc[18235](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:34:25 dist-test-slave-p29l krb5kdc[18235](info): set up 2 sockets
Sep 29 20:34:25 dist-test-slave-p29l krb5kdc[18235](info): commencing operation
krb5kdc: starting...
W20260929 20:34:26.224159 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:34:26 dist-test-slave-p29l krb5kdc[18235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:34:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:34:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:34:26.504186 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33811
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:34:26.862771 18248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:34:26.863122 18248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:34:26.863224 18248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:34:26.870754 18248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:34:26.870886 18248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:34:26.870941 18248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:34:26.871002 18248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:34:26.871048 18248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:34:26.881536 18248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:34:26.882884 18248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:34:26.884244 18248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:34:26.897909 18253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:26.897926 18256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:26.897949 18254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:34:26.898875 18248 server_base.cc:1034] running on GCE node
I20260929 20:34:26.899923 18248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:34:26.901629 18248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:34:26.902928 18248 hybrid_clock.cc:648] HybridClock initialized: now 1790714066902891 us; error 49 us; skew 500 ppm
I20260929 20:34:26.903348 18248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:34:26 dist-test-slave-p29l krb5kdc[18235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714066, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:34:26.907076 18248 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:34:26.908893 18248 webserver.cc:460] Webserver started at http://127.0.0.1:45411/ using document root <none> and password file <none>
I20260929 20:34:26.909646 18248 fs_manager.cc:359] Metadata directory not provided
I20260929 20:34:26.909759 18248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:34:26.910066 18248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:34:26.913727 18248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "29b1ea1375b84ff39a749aed90bd4798"
format_stamp: "Formatted at 2026-09-29 20:34:26 on dist-test-slave-p29l"
server_key: "828f02b9f2ee6bc8c5e7958165a8e26a"
server_key_iv: "7ca62e4c260f46ba815a3f15cb87927d"
server_key_version: "encryptionkey@0"
I20260929 20:34:26.914664 18248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "29b1ea1375b84ff39a749aed90bd4798"
format_stamp: "Formatted at 2026-09-29 20:34:26 on dist-test-slave-p29l"
server_key: "828f02b9f2ee6bc8c5e7958165a8e26a"
server_key_iv: "7ca62e4c260f46ba815a3f15cb87927d"
server_key_version: "encryptionkey@0"
I20260929 20:34:26.920902 18248 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260929 20:34:26.924988 18263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:26.926644 18248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260929 20:34:26.926823 18248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "29b1ea1375b84ff39a749aed90bd4798"
format_stamp: "Formatted at 2026-09-29 20:34:26 on dist-test-slave-p29l"
server_key: "828f02b9f2ee6bc8c5e7958165a8e26a"
server_key_iv: "7ca62e4c260f46ba815a3f15cb87927d"
server_key_version: "encryptionkey@0"
I20260929 20:34:26.927035 18248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:34:26.934155 18248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:34:26.939258 18248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:34:26.939567 18248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:34:26.960103 18248 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33811
I20260929 20:34:26.960188 18314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33811 every 8 connection(s)
I20260929 20:34:26.961621 18248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:34:26.962431 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 18248
I20260929 20:34:26.962642 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:34:26.963260 29650 external_mini_cluster.cc:1428] Setting key a8a52893d8c441e2efcdbfab4f82c840
I20260929 20:34:26.966138 18315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:34:26.973378 18318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:26.965467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37217 (local address 127.0.0.1:33811)
0929 20:34:26.966486 (+ 1019us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:26.966502 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:26.966878 (+ 376us) server_negotiation.cc:407] Connection header received
0929 20:34:26.969230 (+ 2352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:26.969258 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:26.969737 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:26.970220 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:26.971273 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:26.971288 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:26.971308 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:34:26.971325 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:26.971412 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:34:26.971447 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:26.971674 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:26.971829 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 20:34:26.972043 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":723,"thread_start_us":416,"threads_started":1}
I20260929 20:34:26.977574 29650 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:34:26.980599 18315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b1ea1375b84ff39a749aed90bd4798: Bootstrap starting.
I20260929 20:34:26.984575 18315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29b1ea1375b84ff39a749aed90bd4798: Neither blocks nor log segments found. Creating new log.
I20260929 20:34:26.985023 18318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:26.982132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35813 (local address 127.0.0.1:33811)
0929 20:34:26.982329 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:26.982337 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:26.982787 (+ 450us) server_negotiation.cc:407] Connection header received
0929 20:34:26.983003 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:26.983008 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:26.983069 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:26.983174 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:26.984292 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:26.984297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:26.984302 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:34:26.984328 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:26.984374 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:34:26.984391 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:26.984520 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:26.984641 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 20:34:26.984706 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 20:34:26.986264 18315 log.cc:824] T 00000000000000000000000000000000 P 29b1ea1375b84ff39a749aed90bd4798: Log is configured to *not* fsync() on all Append() calls
I20260929 20:34:26.989480 18315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b1ea1375b84ff39a749aed90bd4798: No bootstrap required, opened a new log
I20260929 20:34:26.997538 18315 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29b1ea1375b84ff39a749aed90bd4798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b1ea1375b84ff39a749aed90bd4798" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33811 } }
I20260929 20:34:26.997977 18315 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29b1ea1375b84ff39a749aed90bd4798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:34:26.998055 18315 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29b1ea1375b84ff39a749aed90bd4798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b1ea1375b84ff39a749aed90bd4798, State: Initialized, Role: FOLLOWER
I20260929 20:34:26.998678 18315 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29b1ea1375b84ff39a749aed90bd4798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b1ea1375b84ff39a749aed90bd4798" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33811 } }
I20260929 20:34:26.998904 18315 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29b1ea1375b84ff39a749aed90bd4798 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:34:26.999008 18315 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29b1ea1375b84ff39a749aed90bd4798 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:34:26.999166 18315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29b1ea1375b84ff39a749aed90bd4798 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:34:27.001664 18315 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29b1ea1375b84ff39a749aed90bd4798 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b1ea1375b84ff39a749aed90bd4798" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33811 } }
I20260929 20:34:27.002305 18315 leader_election.cc:302] T 00000000000000000000000000000000 P 29b1ea1375b84ff39a749aed90bd4798 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29b1ea1375b84ff39a749aed90bd4798; no voters:
I20260929 20:34:27.003072 18315 leader_election.cc:288] T 00000000000000000000000000000000 P 29b1ea1375b84ff39a749aed90bd4798 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:34:27.003194 18325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29b1ea1375b84ff39a749aed90bd4798 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:34:27.004045 18325 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29b1ea1375b84ff39a749aed90bd4798 [term 1 LEADER]: Becoming Leader. State: Replica: 29b1ea1375b84ff39a749aed90bd4798, State: Running, Role: LEADER
I20260929 20:34:27.004633 18325 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29b1ea1375b84ff39a749aed90bd4798 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b1ea1375b84ff39a749aed90bd4798" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33811 } }
I20260929 20:34:27.005683 18315 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29b1ea1375b84ff39a749aed90bd4798 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:34:27.007774 18326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b1ea1375b84ff39a749aed90bd4798 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29b1ea1375b84ff39a749aed90bd4798" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b1ea1375b84ff39a749aed90bd4798" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33811 } } }
I20260929 20:34:27.008276 18326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b1ea1375b84ff39a749aed90bd4798 [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:27.008858 18327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b1ea1375b84ff39a749aed90bd4798 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29b1ea1375b84ff39a749aed90bd4798. Latest consensus state: current_term: 1 leader_uuid: "29b1ea1375b84ff39a749aed90bd4798" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b1ea1375b84ff39a749aed90bd4798" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33811 } } }
I20260929 20:34:27.009155 18327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b1ea1375b84ff39a749aed90bd4798 [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:27.010681 18334 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:34:27.015936 18334 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:34:27.026232 18334 catalog_manager.cc:1347] Generated new cluster ID: 94c35c8949f84b538c4afd8d08fffe05
I20260929 20:34:27.026420 18334 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:34:27.039775 18334 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:34:27.040373 18334 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:34:27.049206 18334 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29b1ea1375b84ff39a749aed90bd4798: Generated new TSK 0
I20260929 20:34:27.050411 18334 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:34:27.058612 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 18248
W20260929 20:34:27.060475 18310 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37217
W20260929 20:34:27.061345 29650 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 18248: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:34:27.061563 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 18248
2026-09-29T20:34:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1999 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:34:27 dist-test-slave-p29l krb5kdc[18346](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:34:27 dist-test-slave-p29l krb5kdc[18346](info): set up 2 sockets
Sep 29 20:34:27 dist-test-slave-p29l krb5kdc[18346](info): commencing operation
krb5kdc: starting...
W20260929 20:34:28.232457 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:34:28 dist-test-slave-p29l krb5kdc[18346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:34:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:34:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:34:28.507464 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41111
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:34:28.855063 18359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:34:28.855434 18359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:34:28.855593 18359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:34:28.863116 18359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:34:28.863289 18359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:34:28.863399 18359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:34:28.863507 18359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:34:28.863592 18359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:34:28.874037 18359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:34:28.875499 18359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:34:28.876981 18359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:34:28.889684 18364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:28.889770 18366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:28.889775 18368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:34:28.890936 18359 server_base.cc:1034] running on GCE node
I20260929 20:34:28.891853 18359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:34:28.893491 18359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:34:28.894752 18359 hybrid_clock.cc:648] HybridClock initialized: now 1790714068894711 us; error 45 us; skew 500 ppm
I20260929 20:34:28.895150 18359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:34:28 dist-test-slave-p29l krb5kdc[18346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714068, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:34:28.898582 18359 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:34:28.900398 18359 webserver.cc:460] Webserver started at http://127.0.0.1:36649/ using document root <none> and password file <none>
I20260929 20:34:28.901196 18359 fs_manager.cc:359] Metadata directory not provided
I20260929 20:34:28.901357 18359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:34:28.901672 18359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:34:28.904997 18359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "5fe37d156cd14bffba3f9b625ff50930"
format_stamp: "Formatted at 2026-09-29 20:34:28 on dist-test-slave-p29l"
server_key: "9461727f0736838a32cc354163d24f21"
server_key_iv: "584b1a01ada8021717df700ce5cdada3"
server_key_version: "encryptionkey@0"
I20260929 20:34:28.905823 18359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "5fe37d156cd14bffba3f9b625ff50930"
format_stamp: "Formatted at 2026-09-29 20:34:28 on dist-test-slave-p29l"
server_key: "9461727f0736838a32cc354163d24f21"
server_key_iv: "584b1a01ada8021717df700ce5cdada3"
server_key_version: "encryptionkey@0"
I20260929 20:34:28.911365 18359 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 20:34:28.915385 18375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:28.917174 18359 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:34:28.917428 18359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "5fe37d156cd14bffba3f9b625ff50930"
format_stamp: "Formatted at 2026-09-29 20:34:28 on dist-test-slave-p29l"
server_key: "9461727f0736838a32cc354163d24f21"
server_key_iv: "584b1a01ada8021717df700ce5cdada3"
server_key_version: "encryptionkey@0"
I20260929 20:34:28.917694 18359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:34:28.952880 18359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:34:28.957849 18359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:34:28.958197 18359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:34:28.979146 18359 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41111
I20260929 20:34:28.979179 18427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41111 every 8 connection(s)
I20260929 20:34:28.980728 18359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:34:28.985152 18428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:34:28.985165 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 18359
I20260929 20:34:28.985329 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:34:28.985975 29650 external_mini_cluster.cc:1428] Setting key be4b58552d1ca9a018e61f6b49f8650b
I20260929 20:34:28.999676 18428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fe37d156cd14bffba3f9b625ff50930: Bootstrap starting.
Sep 29 20:34:28 dist-test-slave-p29l krb5kdc[18346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 20:34:29.003882 18428 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fe37d156cd14bffba3f9b625ff50930: Neither blocks nor log segments found. Creating new log.
I20260929 20:34:29.005173 18428 log.cc:824] T 00000000000000000000000000000000 P 5fe37d156cd14bffba3f9b625ff50930: Log is configured to *not* fsync() on all Append() calls
I20260929 20:34:29.008526 18428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fe37d156cd14bffba3f9b625ff50930: No bootstrap required, opened a new log
I20260929 20:34:29.009995 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:28.988103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.159:40377 (local address 127.0.0.1:41111)
0929 20:34:28.989038 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:28.989049 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:28.989094 (+ 45us) server_negotiation.cc:407] Connection header received
0929 20:34:28.990683 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:28.990706 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:28.991145 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:28.991577 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:28.993202 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:28.994662 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:28.996043 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:28.996790 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:28.997028 (+ 238us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:29.002034 (+ 5006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:29.002043 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:29.002062 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:29.002111 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:29.005500 (+ 3389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:29.006370 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:29.006380 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:29.006387 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:29.006557 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:29.007069 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:29.007072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:29.007074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:29.007638 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:29.008013 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:29.008532 (+ 519us) server_negotiation.cc:299] Negotiation successful
0929 20:34:29.008923 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":422,"threads_started":1}
I20260929 20:34:29.014246 29650 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:34:29.019677 18428 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fe37d156cd14bffba3f9b625ff50930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe37d156cd14bffba3f9b625ff50930" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41111 } }
I20260929 20:34:29.020356 18428 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fe37d156cd14bffba3f9b625ff50930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:34:29.020557 18428 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fe37d156cd14bffba3f9b625ff50930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fe37d156cd14bffba3f9b625ff50930, State: Initialized, Role: FOLLOWER
I20260929 20:34:29.021411 18428 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fe37d156cd14bffba3f9b625ff50930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe37d156cd14bffba3f9b625ff50930" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41111 } }
I20260929 20:34:29.021806 18428 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fe37d156cd14bffba3f9b625ff50930 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:34:29.021979 18428 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fe37d156cd14bffba3f9b625ff50930 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:34:29.022194 18428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fe37d156cd14bffba3f9b625ff50930 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:34:29.025797 18428 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fe37d156cd14bffba3f9b625ff50930 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe37d156cd14bffba3f9b625ff50930" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41111 } }
I20260929 20:34:29.026517 18428 leader_election.cc:302] T 00000000000000000000000000000000 P 5fe37d156cd14bffba3f9b625ff50930 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fe37d156cd14bffba3f9b625ff50930; no voters:
I20260929 20:34:29.027235 18428 leader_election.cc:288] T 00000000000000000000000000000000 P 5fe37d156cd14bffba3f9b625ff50930 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:34:29.027401 18438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fe37d156cd14bffba3f9b625ff50930 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:34:29.028998 18438 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fe37d156cd14bffba3f9b625ff50930 [term 1 LEADER]: Becoming Leader. State: Replica: 5fe37d156cd14bffba3f9b625ff50930, State: Running, Role: LEADER
I20260929 20:34:29.029619 18438 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fe37d156cd14bffba3f9b625ff50930 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe37d156cd14bffba3f9b625ff50930" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41111 } }
I20260929 20:34:29.030251 18428 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fe37d156cd14bffba3f9b625ff50930 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:34:29.033273 18440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fe37d156cd14bffba3f9b625ff50930 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fe37d156cd14bffba3f9b625ff50930. Latest consensus state: current_term: 1 leader_uuid: "5fe37d156cd14bffba3f9b625ff50930" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe37d156cd14bffba3f9b625ff50930" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41111 } } }
I20260929 20:34:29.033842 18440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fe37d156cd14bffba3f9b625ff50930 [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:29.034607 18439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fe37d156cd14bffba3f9b625ff50930 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fe37d156cd14bffba3f9b625ff50930" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe37d156cd14bffba3f9b625ff50930" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41111 } } }
I20260929 20:34:29.034983 18446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:34:29.035058 18439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fe37d156cd14bffba3f9b625ff50930 [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:29.040910 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:29.020225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.159:38541 (local address 127.0.0.1:41111)
0929 20:34:29.020460 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:29.020470 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:29.021041 (+ 571us) server_negotiation.cc:407] Connection header received
0929 20:34:29.021241 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:29.021245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:29.021352 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:29.021493 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:29.023907 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:29.024708 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:29.026673 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:29.027339 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:29.027519 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:29.031319 (+ 3800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:29.031327 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:29.031331 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:29.031377 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:29.035352 (+ 3975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:29.036904 (+ 1552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:29.036914 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:29.036925 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:29.037142 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:29.037989 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:29.037998 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:29.038005 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:29.038658 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:29.039047 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:29.040017 (+ 970us) server_negotiation.cc:299] Negotiation successful
0929 20:34:29.040313 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 20:34:29.052322 18446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:34:29.065551 18446 catalog_manager.cc:1347] Generated new cluster ID: 6056919946c943a89e5ae84d456d9164
I20260929 20:34:29.065711 18446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:34:29.077430 18446 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:34:29.079459 18446 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:34:29.089226 18446 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fe37d156cd14bffba3f9b625ff50930: Generated new TSK 0
I20260929 20:34:29.090365 18446 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:34:29.108970 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 18359
I20260929 20:34:29.110500 18423 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:34:29.207007 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 18359
W20260929 20:34:29.225616 18344 connection.cc:535] client connection to 127.0.0.1:41111 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41111): Connection reset by peer (error 104)
2026-09-29T20:34:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2145 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:34:29 dist-test-slave-p29l krb5kdc[18460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:34:29 dist-test-slave-p29l krb5kdc[18460](info): set up 2 sockets
Sep 29 20:34:29 dist-test-slave-p29l krb5kdc[18460](info): commencing operation
krb5kdc: starting...
W20260929 20:34:30.335777 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:34:30 dist-test-slave-p29l krb5kdc[18460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:34:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:34:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:34:30.619107 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38565
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:34:30.973706 18474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:34:30.974041 18474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:34:30.974227 18474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:34:30.982100 18474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:34:30.982249 18474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:34:30.982379 18474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:34:30.982478 18474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:34:30.982571 18474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:34:30.993090 18474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:34:30.994532 18474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:34:30.995980 18474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:34:31.008664 18482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:31.008635 18480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:31.008970 18479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:34:31.009946 18474 server_base.cc:1034] running on GCE node
I20260929 20:34:31.010823 18474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:34:31.012563 18474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:34:31.013823 18474 hybrid_clock.cc:648] HybridClock initialized: now 1790714071013781 us; error 31 us; skew 500 ppm
I20260929 20:34:31.014279 18474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:34:31 dist-test-slave-p29l krb5kdc[18460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:34:31.023648 18474 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:34:31.025619 18474 webserver.cc:460] Webserver started at http://127.0.0.1:42495/ using document root <none> and password file <none>
I20260929 20:34:31.026453 18474 fs_manager.cc:359] Metadata directory not provided
I20260929 20:34:31.026619 18474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:34:31.026991 18474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:34:31.030449 18474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "f31f28dbc438401d9fcf08cc0b2e8718"
format_stamp: "Formatted at 2026-09-29 20:34:31 on dist-test-slave-p29l"
server_key: "6e830855092af4bae95972fed426316f"
server_key_iv: "f90c45b1fc00b7c2752b936bd2a02ec9"
server_key_version: "encryptionkey@0"
I20260929 20:34:31.031320 18474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "f31f28dbc438401d9fcf08cc0b2e8718"
format_stamp: "Formatted at 2026-09-29 20:34:31 on dist-test-slave-p29l"
server_key: "6e830855092af4bae95972fed426316f"
server_key_iv: "f90c45b1fc00b7c2752b936bd2a02ec9"
server_key_version: "encryptionkey@0"
I20260929 20:34:31.037145 18474 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260929 20:34:31.041083 18489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:31.042683 18474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:34:31.042907 18474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "f31f28dbc438401d9fcf08cc0b2e8718"
format_stamp: "Formatted at 2026-09-29 20:34:31 on dist-test-slave-p29l"
server_key: "6e830855092af4bae95972fed426316f"
server_key_iv: "f90c45b1fc00b7c2752b936bd2a02ec9"
server_key_version: "encryptionkey@0"
I20260929 20:34:31.043141 18474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:34:31.061650 18474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:34:31.067045 18474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:34:31.067379 18474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:34:31.089293 18474 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38565
I20260929 20:34:31.089308 18541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38565 every 8 connection(s)
I20260929 20:34:31.090919 18474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:34:31.095111 18542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:34:31.097188 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 18474
I20260929 20:34:31.097386 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:34:31.097965 29650 external_mini_cluster.cc:1428] Setting key 44a9227f2300de90c37358d4fe0c1b45
I20260929 20:34:31.108744 18542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f31f28dbc438401d9fcf08cc0b2e8718: Bootstrap starting.
I20260929 20:34:31.112746 18542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f31f28dbc438401d9fcf08cc0b2e8718: Neither blocks nor log segments found. Creating new log.
I20260929 20:34:31.113883 18545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:31.100373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.159:52377 (local address 127.0.0.1:38565)
0929 20:34:31.101349 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:31.101363 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:31.101419 (+ 56us) server_negotiation.cc:407] Connection header received
0929 20:34:31.103398 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:31.103426 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:31.103978 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:31.104354 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:31.105098 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:31.106551 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:31.108570 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:31.109497 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:31.109664 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:31.111076 (+ 1412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:31.111089 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:31.111109 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:34:31.111737 (+ 628us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:31.111913 (+ 176us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:34:31.111956 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:31.112126 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:31.112430 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 20:34:31.112655 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":439,"threads_started":1}
I20260929 20:34:31.113940 18542 log.cc:824] T 00000000000000000000000000000000 P f31f28dbc438401d9fcf08cc0b2e8718: Log is configured to *not* fsync() on all Append() calls
I20260929 20:34:31.117405 18542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f31f28dbc438401d9fcf08cc0b2e8718: No bootstrap required, opened a new log
I20260929 20:34:31.118674 29650 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:34:31.128490 18542 raft_consensus.cc:348] T 00000000000000000000000000000000 P f31f28dbc438401d9fcf08cc0b2e8718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f31f28dbc438401d9fcf08cc0b2e8718" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38565 } }
I20260929 20:34:31.129230 18542 raft_consensus.cc:374] T 00000000000000000000000000000000 P f31f28dbc438401d9fcf08cc0b2e8718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:34:31.129379 18542 raft_consensus.cc:729] T 00000000000000000000000000000000 P f31f28dbc438401d9fcf08cc0b2e8718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f31f28dbc438401d9fcf08cc0b2e8718, State: Initialized, Role: FOLLOWER
I20260929 20:34:31.130141 18542 consensus_queue.cc:260] T 00000000000000000000000000000000 P f31f28dbc438401d9fcf08cc0b2e8718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f31f28dbc438401d9fcf08cc0b2e8718" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38565 } }
I20260929 20:34:31.130460 18542 raft_consensus.cc:388] T 00000000000000000000000000000000 P f31f28dbc438401d9fcf08cc0b2e8718 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:34:31.130627 18542 raft_consensus.cc:482] T 00000000000000000000000000000000 P f31f28dbc438401d9fcf08cc0b2e8718 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:34:31.130833 18542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f31f28dbc438401d9fcf08cc0b2e8718 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:34:31.132426 18545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:31.124396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.159:60497 (local address 127.0.0.1:38565)
0929 20:34:31.124611 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:31.124622 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:31.125181 (+ 559us) server_negotiation.cc:407] Connection header received
0929 20:34:31.125396 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:31.125401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:31.125486 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:31.125573 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:31.126162 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:31.126971 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:31.129074 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:31.129796 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:31.129953 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:31.131439 (+ 1486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:31.131446 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:31.131452 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:34:31.131483 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:31.131554 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:34:31.131579 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:31.131716 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:31.131988 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 20:34:31.132065 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:34:31.134300 18542 raft_consensus.cc:504] T 00000000000000000000000000000000 P f31f28dbc438401d9fcf08cc0b2e8718 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f31f28dbc438401d9fcf08cc0b2e8718" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38565 } }
I20260929 20:34:31.134824 18542 leader_election.cc:302] T 00000000000000000000000000000000 P f31f28dbc438401d9fcf08cc0b2e8718 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f31f28dbc438401d9fcf08cc0b2e8718; no voters:
I20260929 20:34:31.135429 18542 leader_election.cc:288] T 00000000000000000000000000000000 P f31f28dbc438401d9fcf08cc0b2e8718 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:34:31.135650 18552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f31f28dbc438401d9fcf08cc0b2e8718 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:34:31.137331 18552 raft_consensus.cc:686] T 00000000000000000000000000000000 P f31f28dbc438401d9fcf08cc0b2e8718 [term 1 LEADER]: Becoming Leader. State: Replica: f31f28dbc438401d9fcf08cc0b2e8718, State: Running, Role: LEADER
I20260929 20:34:31.137769 18542 sys_catalog.cc:564] T 00000000000000000000000000000000 P f31f28dbc438401d9fcf08cc0b2e8718 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:34:31.138063 18552 consensus_queue.cc:237] T 00000000000000000000000000000000 P f31f28dbc438401d9fcf08cc0b2e8718 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f31f28dbc438401d9fcf08cc0b2e8718" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38565 } }
I20260929 20:34:31.140974 18554 sys_catalog.cc:455] T 00000000000000000000000000000000 P f31f28dbc438401d9fcf08cc0b2e8718 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f31f28dbc438401d9fcf08cc0b2e8718. Latest consensus state: current_term: 1 leader_uuid: "f31f28dbc438401d9fcf08cc0b2e8718" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f31f28dbc438401d9fcf08cc0b2e8718" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38565 } } }
I20260929 20:34:31.141412 18554 sys_catalog.cc:458] T 00000000000000000000000000000000 P f31f28dbc438401d9fcf08cc0b2e8718 [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:31.141996 18553 sys_catalog.cc:455] T 00000000000000000000000000000000 P f31f28dbc438401d9fcf08cc0b2e8718 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f31f28dbc438401d9fcf08cc0b2e8718" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f31f28dbc438401d9fcf08cc0b2e8718" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38565 } } }
I20260929 20:34:31.142429 18553 sys_catalog.cc:458] T 00000000000000000000000000000000 P f31f28dbc438401d9fcf08cc0b2e8718 [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:31.144138 18560 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:34:31.150098 18560 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:34:31.160614 18560 catalog_manager.cc:1347] Generated new cluster ID: ab4fe26ba050408c936b681c3b1f075f
I20260929 20:34:31.160750 18560 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:34:31.178915 18560 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:34:31.180425 18560 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:34:31.190815 18560 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f31f28dbc438401d9fcf08cc0b2e8718: Generated new TSK 0
I20260929 20:34:31.191779 18560 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:34:31.202904 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 18474
W20260929 20:34:31.204633 18536 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.159:52377
W20260929 20:34:31.205708 29650 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 18474: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:34:31.205883 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 18474
W20260929 20:34:31.224289 18458 connection.cc:535] client connection to 127.0.0.1:38565 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38565): Connection reset by peer (error 104)
2026-09-29T20:34:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1998 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:34:31 dist-test-slave-p29l krb5kdc[18573](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:34:31 dist-test-slave-p29l krb5kdc[18573](info): set up 2 sockets
Sep 29 20:34:31 dist-test-slave-p29l krb5kdc[18573](info): commencing operation
krb5kdc: starting...
W20260929 20:34:32.331923 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:34:32 dist-test-slave-p29l krb5kdc[18573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:34:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:34:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:34:32.611693 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43659
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:34:32.967825 18586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:34:32.968140 18586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:34:32.968310 18586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:34:32.976275 18586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:34:32.976451 18586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:34:32.976543 18586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:34:32.976639 18586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:34:32.976732 18586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:34:32.987109 18586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:34:32.988628 18586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:34:32.989998 18586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:34:33.002688 18591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:33.002696 18594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:33.002599 18592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:34:33.003630 18586 server_base.cc:1034] running on GCE node
I20260929 20:34:33.004597 18586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:34:33.006260 18586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:34:33.007524 18586 hybrid_clock.cc:648] HybridClock initialized: now 1790714073007470 us; error 48 us; skew 500 ppm
I20260929 20:34:33.007964 18586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:34:33 dist-test-slave-p29l krb5kdc[18573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:34:33.011293 18586 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:34:33.013018 18586 webserver.cc:460] Webserver started at http://127.0.0.1:38177/ using document root <none> and password file <none>
I20260929 20:34:33.013768 18586 fs_manager.cc:359] Metadata directory not provided
I20260929 20:34:33.013932 18586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:34:33.014267 18586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:34:33.017575 18586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "54a540780d7a45a1b1e88b4ef9401cc2"
format_stamp: "Formatted at 2026-09-29 20:34:33 on dist-test-slave-p29l"
server_key: "0e0246359ec489a4421365160d91e289"
server_key_iv: "8ca6a657c5538f6378e1db8ce3f982c9"
server_key_version: "encryptionkey@0"
I20260929 20:34:33.018435 18586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "54a540780d7a45a1b1e88b4ef9401cc2"
format_stamp: "Formatted at 2026-09-29 20:34:33 on dist-test-slave-p29l"
server_key: "0e0246359ec489a4421365160d91e289"
server_key_iv: "8ca6a657c5538f6378e1db8ce3f982c9"
server_key_version: "encryptionkey@0"
I20260929 20:34:33.024255 18586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 20:34:33.028381 18601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:33.029881 18586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:34:33.030131 18586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "54a540780d7a45a1b1e88b4ef9401cc2"
format_stamp: "Formatted at 2026-09-29 20:34:33 on dist-test-slave-p29l"
server_key: "0e0246359ec489a4421365160d91e289"
server_key_iv: "8ca6a657c5538f6378e1db8ce3f982c9"
server_key_version: "encryptionkey@0"
I20260929 20:34:33.030364 18586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:34:33.036895 18586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:34:33.041432 18586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:34:33.041771 18586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:34:33.062019 18586 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43659
I20260929 20:34:33.062072 18652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43659 every 8 connection(s)
I20260929 20:34:33.063539 18586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:34:33.067498 18653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:34:33.069420 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 18586
I20260929 20:34:33.069604 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:34:33.070182 29650 external_mini_cluster.cc:1428] Setting key 24286c1fb4eea38e68394f3c27bbc8a3
I20260929 20:34:33.078416 18656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:33.072188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.159:47911 (local address 127.0.0.1:43659)
0929 20:34:33.073099 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:33.073112 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:33.073158 (+ 46us) server_negotiation.cc:407] Connection header received
0929 20:34:33.074801 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:33.074824 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:33.075195 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:33.075546 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:33.076757 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:33.076765 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:33.076778 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:34:33.076794 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:33.076860 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:34:33.076881 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:33.077069 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:33.077179 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 20:34:33.077441 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":401,"threads_started":1}
I20260929 20:34:33.082334 18653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54a540780d7a45a1b1e88b4ef9401cc2: Bootstrap starting.
I20260929 20:34:33.082713 29650 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:34:33.085856 18653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54a540780d7a45a1b1e88b4ef9401cc2: Neither blocks nor log segments found. Creating new log.
I20260929 20:34:33.087081 18653 log.cc:824] T 00000000000000000000000000000000 P 54a540780d7a45a1b1e88b4ef9401cc2: Log is configured to *not* fsync() on all Append() calls
I20260929 20:34:33.090853 18656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:33.087497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.159:46497 (local address 127.0.0.1:43659)
0929 20:34:33.087732 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:33.087741 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:33.088556 (+ 815us) server_negotiation.cc:407] Connection header received
0929 20:34:33.088832 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:33.088837 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:33.088913 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:33.089036 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:33.090119 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:33.090125 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:33.090130 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:34:33.090150 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:33.090204 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:34:33.090227 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:33.090355 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:33.090502 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 20:34:33.090571 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:34:33.090916 18653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54a540780d7a45a1b1e88b4ef9401cc2: No bootstrap required, opened a new log
I20260929 20:34:33.098933 18653 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54a540780d7a45a1b1e88b4ef9401cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54a540780d7a45a1b1e88b4ef9401cc2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43659 } }
I20260929 20:34:33.099459 18653 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54a540780d7a45a1b1e88b4ef9401cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:34:33.099567 18653 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54a540780d7a45a1b1e88b4ef9401cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54a540780d7a45a1b1e88b4ef9401cc2, State: Initialized, Role: FOLLOWER
I20260929 20:34:33.100247 18653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54a540780d7a45a1b1e88b4ef9401cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54a540780d7a45a1b1e88b4ef9401cc2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43659 } }
I20260929 20:34:33.100478 18653 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54a540780d7a45a1b1e88b4ef9401cc2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:34:33.100584 18653 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54a540780d7a45a1b1e88b4ef9401cc2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:34:33.100716 18653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54a540780d7a45a1b1e88b4ef9401cc2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:34:33.102967 18653 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54a540780d7a45a1b1e88b4ef9401cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54a540780d7a45a1b1e88b4ef9401cc2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43659 } }
I20260929 20:34:33.103484 18653 leader_election.cc:302] T 00000000000000000000000000000000 P 54a540780d7a45a1b1e88b4ef9401cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54a540780d7a45a1b1e88b4ef9401cc2; no voters:
I20260929 20:34:33.104154 18653 leader_election.cc:288] T 00000000000000000000000000000000 P 54a540780d7a45a1b1e88b4ef9401cc2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:34:33.104326 18663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54a540780d7a45a1b1e88b4ef9401cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:34:33.105124 18663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54a540780d7a45a1b1e88b4ef9401cc2 [term 1 LEADER]: Becoming Leader. State: Replica: 54a540780d7a45a1b1e88b4ef9401cc2, State: Running, Role: LEADER
I20260929 20:34:33.105786 18663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54a540780d7a45a1b1e88b4ef9401cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54a540780d7a45a1b1e88b4ef9401cc2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43659 } }
I20260929 20:34:33.106781 18653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54a540780d7a45a1b1e88b4ef9401cc2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:34:33.108786 18664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54a540780d7a45a1b1e88b4ef9401cc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54a540780d7a45a1b1e88b4ef9401cc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54a540780d7a45a1b1e88b4ef9401cc2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43659 } } }
I20260929 20:34:33.109169 18664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54a540780d7a45a1b1e88b4ef9401cc2 [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:33.110116 18669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:34:33.109910 18665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54a540780d7a45a1b1e88b4ef9401cc2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54a540780d7a45a1b1e88b4ef9401cc2. Latest consensus state: current_term: 1 leader_uuid: "54a540780d7a45a1b1e88b4ef9401cc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54a540780d7a45a1b1e88b4ef9401cc2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43659 } } }
I20260929 20:34:33.111198 18665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54a540780d7a45a1b1e88b4ef9401cc2 [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:33.117038 18669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:34:33.129172 18669 catalog_manager.cc:1347] Generated new cluster ID: a40584832bb84a41ac360059cbb394be
I20260929 20:34:33.129320 18669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:34:33.163551 18669 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:34:33.164235 18669 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:34:33.172125 18669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54a540780d7a45a1b1e88b4ef9401cc2: Generated new TSK 0
I20260929 20:34:33.173291 18669 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:34:33.183645 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 18586
W20260929 20:34:33.185119 18648 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.159:47911
W20260929 20:34:33.185954 29650 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 18586: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:34:33.186115 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 18586
2026-09-29T20:34:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1980 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:34:33 dist-test-slave-p29l krb5kdc[18685](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:34:33 dist-test-slave-p29l krb5kdc[18685](info): set up 2 sockets
Sep 29 20:34:33 dist-test-slave-p29l krb5kdc[18685](info): commencing operation
krb5kdc: starting...
W20260929 20:34:34.367918 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.143s user 0.007s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:34:34 dist-test-slave-p29l krb5kdc[18685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:34:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:34:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:34:34.642834 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35177
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:34:35.016330 18698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:34:35.016709 18698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:34:35.016889 18698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:34:35.024421 18698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:34:35.024603 18698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:34:35.024710 18698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:34:35.024825 18698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:34:35.024910 18698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:34:35.035523 18698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35177
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:34:35.037041 18698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:34:35.038425 18698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:34:35.051226 18703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:35.051128 18704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:35.051461 18706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:34:35.052201 18698 server_base.cc:1034] running on GCE node
I20260929 20:34:35.053100 18698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:34:35.054596 18698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:34:35.055918 18698 hybrid_clock.cc:648] HybridClock initialized: now 1790714075055781 us; error 84 us; skew 500 ppm
I20260929 20:34:35.056388 18698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:34:35 dist-test-slave-p29l krb5kdc[18685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:34:35.060196 18698 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:34:35.061854 18698 webserver.cc:460] Webserver started at http://127.0.0.1:33741/ using document root <none> and password file <none>
I20260929 20:34:35.062603 18698 fs_manager.cc:359] Metadata directory not provided
I20260929 20:34:35.062722 18698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:34:35.063009 18698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:34:35.066532 18698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "cbd5647491e1433f84217991d84e58ab"
format_stamp: "Formatted at 2026-09-29 20:34:35 on dist-test-slave-p29l"
server_key: "d38273828910e3fea35ee94523019651"
server_key_iv: "4e274202901e16fb0b23a10dfcaea94e"
server_key_version: "encryptionkey@0"
I20260929 20:34:35.067444 18698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "cbd5647491e1433f84217991d84e58ab"
format_stamp: "Formatted at 2026-09-29 20:34:35 on dist-test-slave-p29l"
server_key: "d38273828910e3fea35ee94523019651"
server_key_iv: "4e274202901e16fb0b23a10dfcaea94e"
server_key_version: "encryptionkey@0"
I20260929 20:34:35.073951 18698 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 20:34:35.078047 18713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:35.079600 18698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:34:35.079897 18698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "cbd5647491e1433f84217991d84e58ab"
format_stamp: "Formatted at 2026-09-29 20:34:35 on dist-test-slave-p29l"
server_key: "d38273828910e3fea35ee94523019651"
server_key_iv: "4e274202901e16fb0b23a10dfcaea94e"
server_key_version: "encryptionkey@0"
I20260929 20:34:35.080149 18698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:34:35.097839 18698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:34:35.103019 18698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:34:35.103302 18698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:34:35.124230 18698 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35177
I20260929 20:34:35.124253 18764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35177 every 8 connection(s)
I20260929 20:34:35.125808 18698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:34:35.129931 18765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:34:35.131445 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 18698
I20260929 20:34:35.131639 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:34:35.132304 29650 external_mini_cluster.cc:1428] Setting key f9a859a8a33ac9d48974c36f092bbc7b
I20260929 20:34:35.145391 18765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbd5647491e1433f84217991d84e58ab: Bootstrap starting.
Sep 29 20:34:35 dist-test-slave-p29l krb5kdc[18685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 20:34:35.149040 18765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbd5647491e1433f84217991d84e58ab: Neither blocks nor log segments found. Creating new log.
I20260929 20:34:35.150473 18765 log.cc:824] T 00000000000000000000000000000000 P cbd5647491e1433f84217991d84e58ab: Log is configured to *not* fsync() on all Append() calls
I20260929 20:34:35.153888 18765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbd5647491e1433f84217991d84e58ab: No bootstrap required, opened a new log
I20260929 20:34:35.157747 18768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:35.134436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.159:60791 (local address 127.0.0.1:35177)
0929 20:34:35.135320 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:35.135333 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:35.135382 (+ 49us) server_negotiation.cc:407] Connection header received
0929 20:34:35.137131 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:35.137160 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:35.137685 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:35.138065 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:35.140082 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:35.142225 (+ 2143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:35.143657 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:35.144789 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:35.145000 (+ 211us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:35.149600 (+ 4600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:35.149612 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:35.149633 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:35.149690 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:35.153161 (+ 3471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:35.154092 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:35.154099 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:35.154106 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:35.154278 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:35.154784 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:35.154788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:35.154790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:35.155338 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:35.155716 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:35.156280 (+ 564us) server_negotiation.cc:299] Negotiation successful
0929 20:34:35.156628 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":390,"threads_started":1}
I20260929 20:34:35.162561 29650 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:34:35.163661 18765 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbd5647491e1433f84217991d84e58ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd5647491e1433f84217991d84e58ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35177 } }
I20260929 20:34:35.164161 18765 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbd5647491e1433f84217991d84e58ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:34:35.164273 18765 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbd5647491e1433f84217991d84e58ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbd5647491e1433f84217991d84e58ab, State: Initialized, Role: FOLLOWER
I20260929 20:34:35.164892 18765 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbd5647491e1433f84217991d84e58ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd5647491e1433f84217991d84e58ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35177 } }
I20260929 20:34:35.165158 18765 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbd5647491e1433f84217991d84e58ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:34:35.165310 18765 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbd5647491e1433f84217991d84e58ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:34:35.165495 18765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbd5647491e1433f84217991d84e58ab [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:34:35.168277 18765 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbd5647491e1433f84217991d84e58ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd5647491e1433f84217991d84e58ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35177 } }
I20260929 20:34:35.168987 18765 leader_election.cc:302] T 00000000000000000000000000000000 P cbd5647491e1433f84217991d84e58ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbd5647491e1433f84217991d84e58ab; no voters:
I20260929 20:34:35.169690 18765 leader_election.cc:288] T 00000000000000000000000000000000 P cbd5647491e1433f84217991d84e58ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:34:35.169826 18775 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbd5647491e1433f84217991d84e58ab [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:34:35.172271 18765 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbd5647491e1433f84217991d84e58ab [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:34:35.176167 18775 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbd5647491e1433f84217991d84e58ab [term 1 LEADER]: Becoming Leader. State: Replica: cbd5647491e1433f84217991d84e58ab, State: Running, Role: LEADER
I20260929 20:34:35.177076 18775 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbd5647491e1433f84217991d84e58ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd5647491e1433f84217991d84e58ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35177 } }
I20260929 20:34:35.180902 18777 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbd5647491e1433f84217991d84e58ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbd5647491e1433f84217991d84e58ab. Latest consensus state: current_term: 1 leader_uuid: "cbd5647491e1433f84217991d84e58ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd5647491e1433f84217991d84e58ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35177 } } }
I20260929 20:34:35.181425 18777 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbd5647491e1433f84217991d84e58ab [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:35.182054 18776 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbd5647491e1433f84217991d84e58ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbd5647491e1433f84217991d84e58ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd5647491e1433f84217991d84e58ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35177 } } }
I20260929 20:34:35.182452 18776 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbd5647491e1433f84217991d84e58ab [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:35.183022 18782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:34:35.191215 18768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:35.167677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.159:41533 (local address 127.0.0.1:35177)
0929 20:34:35.167960 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:35.167966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:35.168346 (+ 380us) server_negotiation.cc:407] Connection header received
0929 20:34:35.168582 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:35.168586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:35.168654 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:35.168741 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:35.170855 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:35.172058 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:35.174101 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:35.175194 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:35.175432 (+ 238us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:35.179784 (+ 4352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:35.179843 (+ 59us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:35.179851 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:35.179913 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:35.184637 (+ 4724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:35.186180 (+ 1543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:35.186190 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:35.186197 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:35.186471 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:35.187408 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:35.187418 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:35.187424 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:35.188094 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:35.188478 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:35.190298 (+ 1820us) server_negotiation.cc:299] Negotiation successful
0929 20:34:35.190595 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 20:34:35.196295 18782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:34:35.215337 18782 catalog_manager.cc:1347] Generated new cluster ID: 9610ed9c91a94d42bd3e1c102b6eca02
I20260929 20:34:35.215492 18782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:34:35.224229 18782 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:34:35.226289 18782 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:34:35.238428 18782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbd5647491e1433f84217991d84e58ab: Generated new TSK 0
I20260929 20:34:35.239738 18782 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:34:35.249212 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 18698
I20260929 20:34:35.250761 18760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:34:35.344224 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 18698
W20260929 20:34:35.362565 18683 connection.cc:535] client connection to 127.0.0.1:35177 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35177): Connection reset by peer (error 104)
2026-09-29T20:34:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2158 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:34:35 dist-test-slave-p29l krb5kdc[18797](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:34:35 dist-test-slave-p29l krb5kdc[18797](info): set up 2 sockets
Sep 29 20:34:35 dist-test-slave-p29l krb5kdc[18797](info): commencing operation
krb5kdc: starting...
W20260929 20:34:36.476032 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:34:36 dist-test-slave-p29l krb5kdc[18797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:34:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:34:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:34:36.754782 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40711
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:34:37.101589 18810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:34:37.101917 18810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:34:37.102088 18810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:34:37.109759 18810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:34:37.109908 18810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:34:37.109990 18810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:34:37.110105 18810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:34:37.110177 18810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:34:37.120761 18810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40711
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:34:37.122210 18810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:34:37.123605 18810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:34:37.136492 18815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:37.136415 18816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:37.136431 18818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:34:37.137554 18810 server_base.cc:1034] running on GCE node
I20260929 20:34:37.138453 18810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:34:37.140172 18810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:34:37.141461 18810 hybrid_clock.cc:648] HybridClock initialized: now 1790714077141417 us; error 43 us; skew 500 ppm
I20260929 20:34:37.141860 18810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:34:37 dist-test-slave-p29l krb5kdc[18797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:34:37.149605 18810 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:34:37.151341 18810 webserver.cc:460] Webserver started at http://127.0.0.1:41039/ using document root <none> and password file <none>
I20260929 20:34:37.152186 18810 fs_manager.cc:359] Metadata directory not provided
I20260929 20:34:37.152338 18810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:34:37.152663 18810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:34:37.155798 18810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "0085cb3b3a7e4de88a2788a5345208b7"
format_stamp: "Formatted at 2026-09-29 20:34:37 on dist-test-slave-p29l"
server_key: "bbb530d6e4390be79b1a008ef69a58d9"
server_key_iv: "6f6597fda33800bf69130fc410824e6b"
server_key_version: "encryptionkey@0"
I20260929 20:34:37.156680 18810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "0085cb3b3a7e4de88a2788a5345208b7"
format_stamp: "Formatted at 2026-09-29 20:34:37 on dist-test-slave-p29l"
server_key: "bbb530d6e4390be79b1a008ef69a58d9"
server_key_iv: "6f6597fda33800bf69130fc410824e6b"
server_key_version: "encryptionkey@0"
I20260929 20:34:37.162170 18810 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:34:37.165984 18826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:37.167479 18810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:34:37.167678 18810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "0085cb3b3a7e4de88a2788a5345208b7"
format_stamp: "Formatted at 2026-09-29 20:34:37 on dist-test-slave-p29l"
server_key: "bbb530d6e4390be79b1a008ef69a58d9"
server_key_iv: "6f6597fda33800bf69130fc410824e6b"
server_key_version: "encryptionkey@0"
I20260929 20:34:37.167960 18810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:34:37.191416 18810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:34:37.196928 18810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:34:37.197279 18810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:34:37.217870 18810 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40711
I20260929 20:34:37.217902 18877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40711 every 8 connection(s)
I20260929 20:34:37.219415 18810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:34:37.223291 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 18810
I20260929 20:34:37.223508 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:34:37.223938 18878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:34:37.224080 29650 external_mini_cluster.cc:1428] Setting key 919f1afcce1321cdb1302aa4dcb072f3
I20260929 20:34:37.238842 18881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:37.225980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.159:55391 (local address 127.0.0.1:40711)
0929 20:34:37.226829 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:37.226846 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:37.226890 (+ 44us) server_negotiation.cc:407] Connection header received
0929 20:34:37.229043 (+ 2153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:37.229070 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:37.229524 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:37.229880 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:37.230572 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:37.232070 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:37.233951 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:37.234653 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:37.234796 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:37.236243 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:37.236250 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:37.236267 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:34:37.236924 (+ 657us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:37.237019 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:34:37.237054 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:37.237183 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:37.237489 (+ 306us) server_negotiation.cc:299] Negotiation successful
0929 20:34:37.237735 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":393,"threads_started":1}
I20260929 20:34:37.239311 18878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0085cb3b3a7e4de88a2788a5345208b7: Bootstrap starting.
I20260929 20:34:37.243125 18878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0085cb3b3a7e4de88a2788a5345208b7: Neither blocks nor log segments found. Creating new log.
I20260929 20:34:37.243203 29650 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:34:37.244280 18878 log.cc:824] T 00000000000000000000000000000000 P 0085cb3b3a7e4de88a2788a5345208b7: Log is configured to *not* fsync() on all Append() calls
I20260929 20:34:37.247794 18878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0085cb3b3a7e4de88a2788a5345208b7: No bootstrap required, opened a new log
I20260929 20:34:37.254774 18881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:37.248185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.159:33789 (local address 127.0.0.1:40711)
0929 20:34:37.248405 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:37.248422 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:37.248761 (+ 339us) server_negotiation.cc:407] Connection header received
0929 20:34:37.249017 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:37.249021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:37.249088 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:37.249210 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:37.249750 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:37.250549 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:37.252006 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:37.252521 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:37.252690 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:37.253964 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:37.253970 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:37.253973 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:34:37.253993 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:37.254050 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:34:37.254066 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:37.254196 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:37.254419 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 20:34:37.254482 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:34:37.256728 18878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0085cb3b3a7e4de88a2788a5345208b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0085cb3b3a7e4de88a2788a5345208b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40711 } }
I20260929 20:34:37.257165 18878 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0085cb3b3a7e4de88a2788a5345208b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:34:37.257284 18878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0085cb3b3a7e4de88a2788a5345208b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0085cb3b3a7e4de88a2788a5345208b7, State: Initialized, Role: FOLLOWER
I20260929 20:34:37.257948 18878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0085cb3b3a7e4de88a2788a5345208b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0085cb3b3a7e4de88a2788a5345208b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40711 } }
I20260929 20:34:37.258216 18878 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0085cb3b3a7e4de88a2788a5345208b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:34:37.258370 18878 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0085cb3b3a7e4de88a2788a5345208b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:34:37.258554 18878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0085cb3b3a7e4de88a2788a5345208b7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:34:37.261193 18878 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0085cb3b3a7e4de88a2788a5345208b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0085cb3b3a7e4de88a2788a5345208b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40711 } }
I20260929 20:34:37.261761 18878 leader_election.cc:302] T 00000000000000000000000000000000 P 0085cb3b3a7e4de88a2788a5345208b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0085cb3b3a7e4de88a2788a5345208b7; no voters:
I20260929 20:34:37.262399 18878 leader_election.cc:288] T 00000000000000000000000000000000 P 0085cb3b3a7e4de88a2788a5345208b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:34:37.262555 18888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0085cb3b3a7e4de88a2788a5345208b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:34:37.263415 18888 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0085cb3b3a7e4de88a2788a5345208b7 [term 1 LEADER]: Becoming Leader. State: Replica: 0085cb3b3a7e4de88a2788a5345208b7, State: Running, Role: LEADER
I20260929 20:34:37.264086 18888 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0085cb3b3a7e4de88a2788a5345208b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0085cb3b3a7e4de88a2788a5345208b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40711 } }
I20260929 20:34:37.264892 18878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0085cb3b3a7e4de88a2788a5345208b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:34:37.267418 18890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0085cb3b3a7e4de88a2788a5345208b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0085cb3b3a7e4de88a2788a5345208b7. Latest consensus state: current_term: 1 leader_uuid: "0085cb3b3a7e4de88a2788a5345208b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0085cb3b3a7e4de88a2788a5345208b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40711 } } }
I20260929 20:34:37.267438 18889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0085cb3b3a7e4de88a2788a5345208b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0085cb3b3a7e4de88a2788a5345208b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0085cb3b3a7e4de88a2788a5345208b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40711 } } }
I20260929 20:34:37.267942 18890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0085cb3b3a7e4de88a2788a5345208b7 [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:37.268003 18889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0085cb3b3a7e4de88a2788a5345208b7 [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:37.270334 18894 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:34:37.276209 18894 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:34:37.286312 18894 catalog_manager.cc:1347] Generated new cluster ID: 3c2378e4e4a74430ae1a2ea36315a7e1
I20260929 20:34:37.286478 18894 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:34:37.312490 18894 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:34:37.313992 18894 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:34:37.322463 18894 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0085cb3b3a7e4de88a2788a5345208b7: Generated new TSK 0
I20260929 20:34:37.323388 18894 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:34:37.336210 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 18810
W20260929 20:34:37.337532 18873 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.159:55391
W20260929 20:34:37.338482 29650 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 18810: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:34:37.338647 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 18810
W20260929 20:34:37.357081 18795 connection.cc:535] client connection to 127.0.0.1:40711 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40711): Connection reset by peer (error 104)
2026-09-29T20:34:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1993 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:34:37 dist-test-slave-p29l krb5kdc[18909](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:34:37 dist-test-slave-p29l krb5kdc[18909](info): set up 2 sockets
Sep 29 20:34:37 dist-test-slave-p29l krb5kdc[18909](info): commencing operation
krb5kdc: starting...
W20260929 20:34:38.495769 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:34:38 dist-test-slave-p29l krb5kdc[18909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:34:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:34:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:34:38.769708 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36023
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:34:39.126036 18922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:34:39.126363 18922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:34:39.126459 18922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:34:39.133981 18922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:34:39.134117 18922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:34:39.134164 18922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:34:39.134218 18922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:34:39.134255 18922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:34:39.144676 18922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36023
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713721201454-29650-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:34:39.146124 18922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:34:39.147526 18922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:34:39.160518 18930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:39.160490 18928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:39.160470 18927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:34:39.161288 18922 server_base.cc:1034] running on GCE node
I20260929 20:34:39.162147 18922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:34:39.163684 18922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:34:39.164952 18922 hybrid_clock.cc:648] HybridClock initialized: now 1790714079164908 us; error 56 us; skew 500 ppm
I20260929 20:34:39.165339 18922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:34:39 dist-test-slave-p29l krb5kdc[18909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:34:39.168628 18922 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:34:39.170235 18922 webserver.cc:460] Webserver started at http://127.0.0.1:38453/ using document root <none> and password file <none>
I20260929 20:34:39.170936 18922 fs_manager.cc:359] Metadata directory not provided
I20260929 20:34:39.171046 18922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:34:39.171322 18922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:34:39.174584 18922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "c4bfe1a9fa4f45da9755d3501ff21bf5"
format_stamp: "Formatted at 2026-09-29 20:34:39 on dist-test-slave-p29l"
server_key: "91d802c4d8a9ec34fb8bf3d086b37333"
server_key_iv: "f9880101b88c47b399d962b324512808"
server_key_version: "encryptionkey@0"
I20260929 20:34:39.175396 18922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "c4bfe1a9fa4f45da9755d3501ff21bf5"
format_stamp: "Formatted at 2026-09-29 20:34:39 on dist-test-slave-p29l"
server_key: "91d802c4d8a9ec34fb8bf3d086b37333"
server_key_iv: "f9880101b88c47b399d962b324512808"
server_key_version: "encryptionkey@0"
I20260929 20:34:39.181418 18922 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260929 20:34:39.185280 18937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:39.186980 18922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:34:39.187207 18922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "c4bfe1a9fa4f45da9755d3501ff21bf5"
format_stamp: "Formatted at 2026-09-29 20:34:39 on dist-test-slave-p29l"
server_key: "91d802c4d8a9ec34fb8bf3d086b37333"
server_key_iv: "f9880101b88c47b399d962b324512808"
server_key_version: "encryptionkey@0"
I20260929 20:34:39.187417 18922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:34:39.193796 18922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:34:39.198535 18922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:34:39.198880 18922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:34:39.219213 18922 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36023
I20260929 20:34:39.219273 18989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36023 every 8 connection(s)
I20260929 20:34:39.220810 18922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:34:39.225250 18990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:34:39.227324 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 18922
I20260929 20:34:39.227523 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:34:39.228070 29650 external_mini_cluster.cc:1428] Setting key bbf228eef283c61ed1a1d9faac995919
I20260929 20:34:39.238003 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:39.230187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.159:39237 (local address 127.0.0.1:36023)
0929 20:34:39.231126 (+ 939us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:39.231149 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:39.231208 (+ 59us) server_negotiation.cc:407] Connection header received
0929 20:34:39.233735 (+ 2527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:39.233770 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:39.234264 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:39.234747 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:39.235833 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:39.235842 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:39.235861 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:34:39.235884 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:39.235965 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:34:39.235998 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:39.236244 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:39.236384 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 20:34:39.236620 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":351,"threads_started":1}
I20260929 20:34:39.240017 18990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4bfe1a9fa4f45da9755d3501ff21bf5: Bootstrap starting.
I20260929 20:34:39.242225 29650 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:34:39.243568 18990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4bfe1a9fa4f45da9755d3501ff21bf5: Neither blocks nor log segments found. Creating new log.
I20260929 20:34:39.244715 18990 log.cc:824] T 00000000000000000000000000000000 P c4bfe1a9fa4f45da9755d3501ff21bf5: Log is configured to *not* fsync() on all Append() calls
I20260929 20:34:39.248312 18990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4bfe1a9fa4f45da9755d3501ff21bf5: No bootstrap required, opened a new log
I20260929 20:34:39.249629 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:39.246970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.159:44091 (local address 127.0.0.1:36023)
0929 20:34:39.247169 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:39.247176 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:39.247550 (+ 374us) server_negotiation.cc:407] Connection header received
0929 20:34:39.247692 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:39.247697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:39.247760 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:39.247939 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:39.248934 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:39.248946 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:39.248950 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:34:39.248967 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:39.249008 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:34:39.249025 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:39.249117 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:39.249262 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 20:34:39.249313 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 20:34:39.257210 18990 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4bfe1a9fa4f45da9755d3501ff21bf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4bfe1a9fa4f45da9755d3501ff21bf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36023 } }
I20260929 20:34:39.257781 18990 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4bfe1a9fa4f45da9755d3501ff21bf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:34:39.257905 18990 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4bfe1a9fa4f45da9755d3501ff21bf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4bfe1a9fa4f45da9755d3501ff21bf5, State: Initialized, Role: FOLLOWER
I20260929 20:34:39.258531 18990 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4bfe1a9fa4f45da9755d3501ff21bf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4bfe1a9fa4f45da9755d3501ff21bf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36023 } }
I20260929 20:34:39.258797 18990 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4bfe1a9fa4f45da9755d3501ff21bf5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:34:39.258950 18990 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4bfe1a9fa4f45da9755d3501ff21bf5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:34:39.259132 18990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4bfe1a9fa4f45da9755d3501ff21bf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:34:39.261570 18990 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4bfe1a9fa4f45da9755d3501ff21bf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4bfe1a9fa4f45da9755d3501ff21bf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36023 } }
I20260929 20:34:39.262220 18990 leader_election.cc:302] T 00000000000000000000000000000000 P c4bfe1a9fa4f45da9755d3501ff21bf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4bfe1a9fa4f45da9755d3501ff21bf5; no voters:
I20260929 20:34:39.262908 18990 leader_election.cc:288] T 00000000000000000000000000000000 P c4bfe1a9fa4f45da9755d3501ff21bf5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:34:39.263049 19000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4bfe1a9fa4f45da9755d3501ff21bf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:34:39.263867 19000 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4bfe1a9fa4f45da9755d3501ff21bf5 [term 1 LEADER]: Becoming Leader. State: Replica: c4bfe1a9fa4f45da9755d3501ff21bf5, State: Running, Role: LEADER
I20260929 20:34:39.264590 19000 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4bfe1a9fa4f45da9755d3501ff21bf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4bfe1a9fa4f45da9755d3501ff21bf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36023 } }
I20260929 20:34:39.265362 18990 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4bfe1a9fa4f45da9755d3501ff21bf5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:34:39.267607 19002 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4bfe1a9fa4f45da9755d3501ff21bf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4bfe1a9fa4f45da9755d3501ff21bf5. Latest consensus state: current_term: 1 leader_uuid: "c4bfe1a9fa4f45da9755d3501ff21bf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4bfe1a9fa4f45da9755d3501ff21bf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36023 } } }
I20260929 20:34:39.267635 19001 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4bfe1a9fa4f45da9755d3501ff21bf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4bfe1a9fa4f45da9755d3501ff21bf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4bfe1a9fa4f45da9755d3501ff21bf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36023 } } }
I20260929 20:34:39.268158 19001 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4bfe1a9fa4f45da9755d3501ff21bf5 [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:39.268158 19002 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4bfe1a9fa4f45da9755d3501ff21bf5 [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:39.269994 19007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:34:39.276074 19007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:34:39.286326 19007 catalog_manager.cc:1347] Generated new cluster ID: 3d20a8024066439794e4aa16f49a7241
I20260929 20:34:39.286486 19007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:34:39.304692 19007 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:34:39.305202 19007 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:34:39.318607 19007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4bfe1a9fa4f45da9755d3501ff21bf5: Generated new TSK 0
I20260929 20:34:39.319904 19007 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:34:39.329074 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 18922
W20260929 20:34:39.330615 18985 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.159:39237
W20260929 20:34:39.331413 29650 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 18922: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:34:39.331602 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 18922
2026-09-29T20:34:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1992 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:34:39 dist-test-slave-p29l krb5kdc[19021](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:34:39 dist-test-slave-p29l krb5kdc[19021](info): set up 2 sockets
Sep 29 20:34:39 dist-test-slave-p29l krb5kdc[19021](info): commencing operation
krb5kdc: starting...
W20260929 20:34:40.464210 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:34:40 dist-test-slave-p29l krb5kdc[19021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:34:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:34:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.244.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
I20260929 20:34:40.743240 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:44643
--webserver_interface=127.28.244.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:35231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:44643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:34:41.092763 19034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:34:41.093225 19034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:34:41.093405 19034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:34:41.101119 19034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:34:41.101316 19034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:34:41.101430 19034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:34:41.101545 19034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:34:41.101640 19034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:34:41.112145 19034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:35231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:44643
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:44643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:34:41.113644 19034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:34:41.115069 19034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:34:41.127748 19042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:41.127748 19040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:41.128273 19039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:34:41.128690 19034 server_base.cc:1034] running on GCE node
I20260929 20:34:41.129598 19034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:34:41.131150 19034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:34:41.132392 19034 hybrid_clock.cc:648] HybridClock initialized: now 1790714081132348 us; error 62 us; skew 500 ppm
I20260929 20:34:41.132783 19034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:34:41 dist-test-slave-p29l krb5kdc[19021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714081, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:34:41.136248 19034 init.cc:377] Logged in from keytab as kudu/127.28.244.190@KRBTEST.COM (short username kudu)
I20260929 20:34:41.137862 19034 webserver.cc:460] Webserver started at http://127.28.244.190:36649/ using document root <none> and password file <none>
I20260929 20:34:41.138661 19034 fs_manager.cc:359] Metadata directory not provided
I20260929 20:34:41.138800 19034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:34:41.139086 19034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:34:41.142234 19034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "69d33244b01c4c90bb338c4cf0df3c8b"
format_stamp: "Formatted at 2026-09-29 20:34:41 on dist-test-slave-p29l"
server_key: "7e5230949d09d027e6e935e0a1bcd39c"
server_key_iv: "a23ccb00b4bbe3a4e37a1f30a28ed7a8"
server_key_version: "encryptionkey@0"
I20260929 20:34:41.143002 19034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "69d33244b01c4c90bb338c4cf0df3c8b"
format_stamp: "Formatted at 2026-09-29 20:34:41 on dist-test-slave-p29l"
server_key: "7e5230949d09d027e6e935e0a1bcd39c"
server_key_iv: "a23ccb00b4bbe3a4e37a1f30a28ed7a8"
server_key_version: "encryptionkey@0"
I20260929 20:34:41.148957 19034 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 20:34:41.152842 19049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:41.154336 19034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:34:41.154551 19034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "69d33244b01c4c90bb338c4cf0df3c8b"
format_stamp: "Formatted at 2026-09-29 20:34:41 on dist-test-slave-p29l"
server_key: "7e5230949d09d027e6e935e0a1bcd39c"
server_key_iv: "a23ccb00b4bbe3a4e37a1f30a28ed7a8"
server_key_version: "encryptionkey@0"
I20260929 20:34:41.154750 19034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:34:41.170523 19034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:34:41.175508 19034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:34:41.175889 19034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:34:41.195775 19034 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:44643
I20260929 20:34:41.195920 19100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:44643 every 8 connection(s)
I20260929 20:34:41.197305 19034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:34:41.200161 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 19034
I20260929 20:34:41.200361 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:34:41.200812 29650 external_mini_cluster.cc:1428] Setting key 54781abeb723fa0dccc31fca8b96f9b6
I20260929 20:34:41.201658 19101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 20:34:41 dist-test-slave-p29l krb5kdc[19021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:34:41.217875 19101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69d33244b01c4c90bb338c4cf0df3c8b: Bootstrap starting.
I20260929 20:34:41.221397 19101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69d33244b01c4c90bb338c4cf0df3c8b: Neither blocks nor log segments found. Creating new log.
I20260929 20:34:41.222790 19101 log.cc:824] T 00000000000000000000000000000000 P 69d33244b01c4c90bb338c4cf0df3c8b: Log is configured to *not* fsync() on all Append() calls
I20260929 20:34:41.226433 19101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69d33244b01c4c90bb338c4cf0df3c8b: No bootstrap required, opened a new log
I20260929 20:34:41.226603 19104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:41.202732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50883 (local address 127.28.244.190:44643)
0929 20:34:41.203604 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:41.203616 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:41.203662 (+ 46us) server_negotiation.cc:407] Connection header received
0929 20:34:41.205368 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:41.205392 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:41.205837 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:41.206220 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:41.208256 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:41.209694 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:41.211611 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:41.212397 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:41.212542 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:41.217866 (+ 5324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:41.217876 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:41.217893 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:41.217938 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:41.221243 (+ 3305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:41.222376 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:41.222384 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:41.222392 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:41.222607 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:41.223178 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:41.223182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:41.223184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:41.224181 (+ 997us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:41.224519 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:41.225054 (+ 535us) server_negotiation.cc:299] Negotiation successful
0929 20:34:41.225413 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":407,"threads_started":1}
I20260929 20:34:41.231108 29650 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:34:41.235705 19101 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69d33244b01c4c90bb338c4cf0df3c8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d33244b01c4c90bb338c4cf0df3c8b" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 44643 } }
I20260929 20:34:41.236398 19101 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69d33244b01c4c90bb338c4cf0df3c8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:34:41.236536 19101 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69d33244b01c4c90bb338c4cf0df3c8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69d33244b01c4c90bb338c4cf0df3c8b, State: Initialized, Role: FOLLOWER
I20260929 20:34:41.237228 19101 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69d33244b01c4c90bb338c4cf0df3c8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d33244b01c4c90bb338c4cf0df3c8b" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 44643 } }
I20260929 20:34:41.237484 19101 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69d33244b01c4c90bb338c4cf0df3c8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:34:41.237641 19101 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69d33244b01c4c90bb338c4cf0df3c8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:34:41.237831 19101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69d33244b01c4c90bb338c4cf0df3c8b [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:34:41.240365 19101 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69d33244b01c4c90bb338c4cf0df3c8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d33244b01c4c90bb338c4cf0df3c8b" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 44643 } }
I20260929 20:34:41.241052 19101 leader_election.cc:302] T 00000000000000000000000000000000 P 69d33244b01c4c90bb338c4cf0df3c8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69d33244b01c4c90bb338c4cf0df3c8b; no voters:
I20260929 20:34:41.241834 19101 leader_election.cc:288] T 00000000000000000000000000000000 P 69d33244b01c4c90bb338c4cf0df3c8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:34:41.242017 19111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69d33244b01c4c90bb338c4cf0df3c8b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:34:41.246306 19111 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69d33244b01c4c90bb338c4cf0df3c8b [term 1 LEADER]: Becoming Leader. State: Replica: 69d33244b01c4c90bb338c4cf0df3c8b, State: Running, Role: LEADER
I20260929 20:34:41.247138 19111 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69d33244b01c4c90bb338c4cf0df3c8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d33244b01c4c90bb338c4cf0df3c8b" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 44643 } }
I20260929 20:34:41.250092 19112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69d33244b01c4c90bb338c4cf0df3c8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69d33244b01c4c90bb338c4cf0df3c8b. Latest consensus state: current_term: 1 leader_uuid: "69d33244b01c4c90bb338c4cf0df3c8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d33244b01c4c90bb338c4cf0df3c8b" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 44643 } } }
I20260929 20:34:41.250761 19112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69d33244b01c4c90bb338c4cf0df3c8b [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:41.251031 19113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69d33244b01c4c90bb338c4cf0df3c8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69d33244b01c4c90bb338c4cf0df3c8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d33244b01c4c90bb338c4cf0df3c8b" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 44643 } } }
I20260929 20:34:41.251276 19101 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69d33244b01c4c90bb338c4cf0df3c8b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:34:41.251422 19113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69d33244b01c4c90bb338c4cf0df3c8b [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:41.260033 19104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:41.235706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55741 (local address 127.28.244.190:44643)
0929 20:34:41.235978 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:41.235987 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:41.236343 (+ 356us) server_negotiation.cc:407] Connection header received
0929 20:34:41.236567 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:41.236573 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:41.236671 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:41.236770 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:41.238524 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:41.239364 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:41.240718 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:41.241408 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:41.241567 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:41.246239 (+ 4672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:41.246253 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:41.246261 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:41.246330 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:41.251360 (+ 5030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:41.252705 (+ 1345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:41.252713 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:41.252720 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:41.252944 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:41.253741 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:41.253749 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:41.253755 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:41.254272 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:41.258389 (+ 4117us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:41.258926 (+ 537us) server_negotiation.cc:299] Negotiation successful
0929 20:34:41.259258 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 20:34:41.262339 19115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:34:41.268951 19115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:34:41.279198 19115 catalog_manager.cc:1347] Generated new cluster ID: 903704d1ba38421480a214c3876dff96
I20260929 20:34:41.279362 19115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:34:41.291939 19115 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:34:41.294029 19115 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:34:41.307044 19115 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69d33244b01c4c90bb338c4cf0df3c8b: Generated new TSK 0
I20260929 20:34:41.308244 19115 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:34:41.321128 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 19034
I20260929 20:34:41.322429 19096 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:34:41.416494 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 19034
W20260929 20:34:41.435043 19019 connection.cc:535] client connection to 127.28.244.190:44643 recv error: Network error: failed to read from TLS socket (remote: 127.28.244.190:44643): Connection reset by peer (error 104)
2026-09-29T20:34:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2085 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:34:41 dist-test-slave-p29l krb5kdc[19133](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:34:41 dist-test-slave-p29l krb5kdc[19133](info): set up 2 sockets
Sep 29 20:34:41 dist-test-slave-p29l krb5kdc[19133](info): commencing operation
krb5kdc: starting...
W20260929 20:34:42.543944 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:34:42 dist-test-slave-p29l krb5kdc[19133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:34:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:34:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.244.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
I20260929 20:34:42.827652 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:38741
--webserver_interface=127.28.244.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:43755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:38741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:34:43.176031 19146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:34:43.176369 19146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:34:43.176465 19146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:34:43.184163 19146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:34:43.184295 19146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:34:43.184343 19146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:34:43.184410 19146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:34:43.184458 19146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:34:43.194944 19146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:43755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:38741
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:38741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:34:43.196377 19146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:34:43.197716 19146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:34:43.211141 19155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:43.211077 19152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:34:43.211467 19146 server_base.cc:1034] running on GCE node
W20260929 20:34:43.211508 19151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:34:43.212623 19146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:34:43.214324 19146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:34:43.215561 19146 hybrid_clock.cc:648] HybridClock initialized: now 1790714083215507 us; error 39 us; skew 500 ppm
I20260929 20:34:43.216097 19146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:34:43 dist-test-slave-p29l krb5kdc[19133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714083, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:34:43.223281 19146 init.cc:377] Logged in from keytab as kudu/127.28.244.190@KRBTEST.COM (short username kudu)
I20260929 20:34:43.225389 19146 webserver.cc:460] Webserver started at http://127.28.244.190:34197/ using document root <none> and password file <none>
I20260929 20:34:43.226233 19146 fs_manager.cc:359] Metadata directory not provided
I20260929 20:34:43.226392 19146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:34:43.226750 19146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:34:43.230181 19146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "a8327e13807a4013b5821b562bd3c343"
format_stamp: "Formatted at 2026-09-29 20:34:43 on dist-test-slave-p29l"
server_key: "ef2200db2e999e9b3ada9989051cadeb"
server_key_iv: "555c98c823afd9e764a6d1bd126b2ca8"
server_key_version: "encryptionkey@0"
I20260929 20:34:43.231096 19146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "a8327e13807a4013b5821b562bd3c343"
format_stamp: "Formatted at 2026-09-29 20:34:43 on dist-test-slave-p29l"
server_key: "ef2200db2e999e9b3ada9989051cadeb"
server_key_iv: "555c98c823afd9e764a6d1bd126b2ca8"
server_key_version: "encryptionkey@0"
I20260929 20:34:43.237306 19146 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 20:34:43.241374 19162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:43.242992 19146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:34:43.243229 19146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "a8327e13807a4013b5821b562bd3c343"
format_stamp: "Formatted at 2026-09-29 20:34:43 on dist-test-slave-p29l"
server_key: "ef2200db2e999e9b3ada9989051cadeb"
server_key_iv: "555c98c823afd9e764a6d1bd126b2ca8"
server_key_version: "encryptionkey@0"
I20260929 20:34:43.243468 19146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:34:43.272080 19146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:34:43.276932 19146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:34:43.277262 19146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:34:43.297312 19146 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:38741
I20260929 20:34:43.297312 19214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:38741 every 8 connection(s)
I20260929 20:34:43.298749 19146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:34:43.302747 19215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:34:43.305603 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 19146
I20260929 20:34:43.305799 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:34:43.306353 29650 external_mini_cluster.cc:1428] Setting key c5082af104b3b4b110f0b3a32f3687c1
I20260929 20:34:43.315609 19215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8327e13807a4013b5821b562bd3c343: Bootstrap starting.
I20260929 20:34:43.318958 19215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8327e13807a4013b5821b562bd3c343: Neither blocks nor log segments found. Creating new log.
I20260929 20:34:43.320446 19215 log.cc:824] T 00000000000000000000000000000000 P a8327e13807a4013b5821b562bd3c343: Log is configured to *not* fsync() on all Append() calls
I20260929 20:34:43.321202 19218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:43.308660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46785 (local address 127.28.244.190:38741)
0929 20:34:43.309583 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:43.309597 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:43.309644 (+ 47us) server_negotiation.cc:407] Connection header received
0929 20:34:43.311417 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:43.311442 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:43.311988 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:43.312411 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:43.313064 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:43.314421 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:43.315898 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:43.316624 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:43.316846 (+ 222us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:43.318180 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:43.318188 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:43.318204 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:34:43.318741 (+ 537us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:43.318823 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:34:43.318853 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:43.319044 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:43.319431 (+ 387us) server_negotiation.cc:299] Negotiation successful
0929 20:34:43.319667 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":391,"threads_started":1}
I20260929 20:34:43.323853 19215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8327e13807a4013b5821b562bd3c343: No bootstrap required, opened a new log
I20260929 20:34:43.325317 29650 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:34:43.334715 19215 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8327e13807a4013b5821b562bd3c343 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8327e13807a4013b5821b562bd3c343" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 38741 } }
I20260929 20:34:43.335249 19215 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8327e13807a4013b5821b562bd3c343 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:34:43.335369 19215 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8327e13807a4013b5821b562bd3c343 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8327e13807a4013b5821b562bd3c343, State: Initialized, Role: FOLLOWER
I20260929 20:34:43.336223 19215 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8327e13807a4013b5821b562bd3c343 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8327e13807a4013b5821b562bd3c343" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 38741 } }
I20260929 20:34:43.336513 19215 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8327e13807a4013b5821b562bd3c343 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:34:43.336691 19215 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8327e13807a4013b5821b562bd3c343 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:34:43.336882 19215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8327e13807a4013b5821b562bd3c343 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:34:43.338083 19218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:43.330874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45275 (local address 127.28.244.190:38741)
0929 20:34:43.331077 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:43.331084 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:43.331534 (+ 450us) server_negotiation.cc:407] Connection header received
0929 20:34:43.331672 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:43.331677 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:43.331761 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:43.331894 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:43.332461 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:43.333703 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:43.335066 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:43.335735 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:43.335916 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:43.337162 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:43.337169 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:43.337174 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:34:43.337202 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:43.337272 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:34:43.337297 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:43.337427 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:43.337625 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 20:34:43.337693 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 20:34:43.340181 19215 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8327e13807a4013b5821b562bd3c343 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8327e13807a4013b5821b562bd3c343" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 38741 } }
I20260929 20:34:43.340687 19215 leader_election.cc:302] T 00000000000000000000000000000000 P a8327e13807a4013b5821b562bd3c343 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8327e13807a4013b5821b562bd3c343; no voters:
I20260929 20:34:43.341284 19215 leader_election.cc:288] T 00000000000000000000000000000000 P a8327e13807a4013b5821b562bd3c343 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:34:43.341452 19225 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8327e13807a4013b5821b562bd3c343 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:34:43.342693 19225 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8327e13807a4013b5821b562bd3c343 [term 1 LEADER]: Becoming Leader. State: Replica: a8327e13807a4013b5821b562bd3c343, State: Running, Role: LEADER
I20260929 20:34:43.343304 19225 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8327e13807a4013b5821b562bd3c343 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8327e13807a4013b5821b562bd3c343" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 38741 } }
I20260929 20:34:43.343964 19215 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8327e13807a4013b5821b562bd3c343 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:34:43.346458 19226 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8327e13807a4013b5821b562bd3c343 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8327e13807a4013b5821b562bd3c343" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8327e13807a4013b5821b562bd3c343" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 38741 } } }
I20260929 20:34:43.346935 19226 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8327e13807a4013b5821b562bd3c343 [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:43.346587 19227 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8327e13807a4013b5821b562bd3c343 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8327e13807a4013b5821b562bd3c343. Latest consensus state: current_term: 1 leader_uuid: "a8327e13807a4013b5821b562bd3c343" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8327e13807a4013b5821b562bd3c343" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 38741 } } }
I20260929 20:34:43.348531 19227 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8327e13807a4013b5821b562bd3c343 [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:43.353235 19232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:34:43.358740 19232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:34:43.371054 19232 catalog_manager.cc:1347] Generated new cluster ID: 4eaeb4c3b3144a66a7959adce7bb22fe
I20260929 20:34:43.371193 19232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:34:43.380228 19232 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:34:43.382144 19232 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:34:43.393388 19232 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8327e13807a4013b5821b562bd3c343: Generated new TSK 0
I20260929 20:34:43.394594 19232 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:34:43.400694 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 19146
W20260929 20:34:43.401865 19210 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46785
W20260929 20:34:43.402740 29650 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 19146: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:34:43.402911 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 19146
W20260929 20:34:43.420838 19131 connection.cc:535] client connection to 127.28.244.190:38741 recv error: Network error: failed to read from TLS socket (remote: 127.28.244.190:38741): Connection reset by peer (error 104)
2026-09-29T20:34:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1985 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:34:43 dist-test-slave-p29l krb5kdc[19246](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:34:43 dist-test-slave-p29l krb5kdc[19246](info): set up 2 sockets
Sep 29 20:34:43 dist-test-slave-p29l krb5kdc[19246](info): commencing operation
krb5kdc: starting...
W20260929 20:34:44.616541 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.176s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:34:44 dist-test-slave-p29l krb5kdc[19246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:34:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:34:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.244.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
I20260929 20:34:44.895982 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:38513
--webserver_interface=127.28.244.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:45171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:38513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:34:45.264644 19260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:34:45.265038 19260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:34:45.265216 19260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:34:45.272796 19260 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:34:45.273006 19260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:34:45.273128 19260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:34:45.273239 19260 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:34:45.273329 19260 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:34:45.283645 19260 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:45171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:38513
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:38513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:34:45.285126 19260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:34:45.286540 19260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:34:45.299690 19265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:45.299959 19266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:45.299959 19268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:34:45.300966 19260 server_base.cc:1034] running on GCE node
I20260929 20:34:45.301875 19260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:34:45.303540 19260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:34:45.304798 19260 hybrid_clock.cc:648] HybridClock initialized: now 1790714085304731 us; error 62 us; skew 500 ppm
I20260929 20:34:45.305186 19260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:34:45 dist-test-slave-p29l krb5kdc[19246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714085, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:34:45.308812 19260 init.cc:377] Logged in from keytab as kudu/127.28.244.190@KRBTEST.COM (short username kudu)
I20260929 20:34:45.310566 19260 webserver.cc:460] Webserver started at http://127.28.244.190:38305/ using document root <none> and password file <none>
I20260929 20:34:45.311348 19260 fs_manager.cc:359] Metadata directory not provided
I20260929 20:34:45.311522 19260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:34:45.311918 19260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:34:45.315354 19260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "36d32d527dc54076aec52befd08c278c"
format_stamp: "Formatted at 2026-09-29 20:34:45 on dist-test-slave-p29l"
server_key: "bc1789bdaa8a5acd9e5a2bf80a940480"
server_key_iv: "6758f59f811cad20598c478915fe9bb2"
server_key_version: "encryptionkey@0"
I20260929 20:34:45.316257 19260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "36d32d527dc54076aec52befd08c278c"
format_stamp: "Formatted at 2026-09-29 20:34:45 on dist-test-slave-p29l"
server_key: "bc1789bdaa8a5acd9e5a2bf80a940480"
server_key_iv: "6758f59f811cad20598c478915fe9bb2"
server_key_version: "encryptionkey@0"
I20260929 20:34:45.321921 19260 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 20:34:45.325948 19275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:45.327445 19260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:34:45.327678 19260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "36d32d527dc54076aec52befd08c278c"
format_stamp: "Formatted at 2026-09-29 20:34:45 on dist-test-slave-p29l"
server_key: "bc1789bdaa8a5acd9e5a2bf80a940480"
server_key_iv: "6758f59f811cad20598c478915fe9bb2"
server_key_version: "encryptionkey@0"
I20260929 20:34:45.327948 19260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:34:45.334547 19260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:34:45.339608 19260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:34:45.340027 19260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:34:45.360389 19260 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:38513
I20260929 20:34:45.360409 19326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:38513 every 8 connection(s)
I20260929 20:34:45.361901 19260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:34:45.364040 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 19260
I20260929 20:34:45.364234 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:34:45.364802 29650 external_mini_cluster.cc:1428] Setting key 963da39780a070e7b47001d220be2eaa
I20260929 20:34:45.366575 19327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:34:45.373283 19331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:45.366783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50397 (local address 127.28.244.190:38513)
0929 20:34:45.367705 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:45.367718 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:45.367761 (+ 43us) server_negotiation.cc:407] Connection header received
0929 20:34:45.369499 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:45.369523 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:45.369895 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:45.370361 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:45.371485 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:45.371491 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:45.371505 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:34:45.371520 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:45.371601 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:34:45.371621 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:45.371903 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:45.372050 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 20:34:45.372256 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":403,"threads_started":1}
I20260929 20:34:45.377365 29650 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:34:45.379935 19327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36d32d527dc54076aec52befd08c278c: Bootstrap starting.
I20260929 20:34:45.383731 19327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36d32d527dc54076aec52befd08c278c: Neither blocks nor log segments found. Creating new log.
I20260929 20:34:45.384543 19331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:45.381912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59727 (local address 127.28.244.190:38513)
0929 20:34:45.382131 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:45.382137 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:45.382413 (+ 276us) server_negotiation.cc:407] Connection header received
0929 20:34:45.382626 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:45.382636 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:45.382696 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:45.382802 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:45.383760 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:45.383765 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:45.383770 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:34:45.383797 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:45.383889 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:34:45.383910 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:45.384031 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:45.384186 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 20:34:45.384260 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 20:34:45.385145 19327 log.cc:824] T 00000000000000000000000000000000 P 36d32d527dc54076aec52befd08c278c: Log is configured to *not* fsync() on all Append() calls
I20260929 20:34:45.388235 19327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36d32d527dc54076aec52befd08c278c: No bootstrap required, opened a new log
I20260929 20:34:45.395902 19327 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36d32d527dc54076aec52befd08c278c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d32d527dc54076aec52befd08c278c" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 38513 } }
I20260929 20:34:45.396355 19327 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36d32d527dc54076aec52befd08c278c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:34:45.396476 19327 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36d32d527dc54076aec52befd08c278c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36d32d527dc54076aec52befd08c278c, State: Initialized, Role: FOLLOWER
I20260929 20:34:45.397069 19327 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36d32d527dc54076aec52befd08c278c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d32d527dc54076aec52befd08c278c" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 38513 } }
I20260929 20:34:45.397323 19327 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36d32d527dc54076aec52befd08c278c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:34:45.397493 19327 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36d32d527dc54076aec52befd08c278c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:34:45.397672 19327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36d32d527dc54076aec52befd08c278c [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:34:45.400080 19327 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36d32d527dc54076aec52befd08c278c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d32d527dc54076aec52befd08c278c" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 38513 } }
I20260929 20:34:45.400678 19327 leader_election.cc:302] T 00000000000000000000000000000000 P 36d32d527dc54076aec52befd08c278c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36d32d527dc54076aec52befd08c278c; no voters:
I20260929 20:34:45.401286 19327 leader_election.cc:288] T 00000000000000000000000000000000 P 36d32d527dc54076aec52befd08c278c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:34:45.401445 19338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36d32d527dc54076aec52befd08c278c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:34:45.402191 19338 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36d32d527dc54076aec52befd08c278c [term 1 LEADER]: Becoming Leader. State: Replica: 36d32d527dc54076aec52befd08c278c, State: Running, Role: LEADER
I20260929 20:34:45.402748 19338 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36d32d527dc54076aec52befd08c278c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d32d527dc54076aec52befd08c278c" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 38513 } }
I20260929 20:34:45.403636 19327 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36d32d527dc54076aec52befd08c278c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:34:45.406011 19340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36d32d527dc54076aec52befd08c278c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36d32d527dc54076aec52befd08c278c. Latest consensus state: current_term: 1 leader_uuid: "36d32d527dc54076aec52befd08c278c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d32d527dc54076aec52befd08c278c" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 38513 } } }
I20260929 20:34:45.405992 19339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36d32d527dc54076aec52befd08c278c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36d32d527dc54076aec52befd08c278c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d32d527dc54076aec52befd08c278c" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 38513 } } }
I20260929 20:34:45.406490 19340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36d32d527dc54076aec52befd08c278c [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:45.406539 19339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36d32d527dc54076aec52befd08c278c [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:45.413595 19344 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:34:45.419469 19344 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:34:45.439690 19344 catalog_manager.cc:1347] Generated new cluster ID: 0b53fb9653784034842968c7f0f81917
I20260929 20:34:45.439975 19344 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:34:45.457978 19344 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:34:45.458546 19344 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:34:45.470433 19344 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36d32d527dc54076aec52befd08c278c: Generated new TSK 0
I20260929 20:34:45.471477 19344 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:34:45.490592 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 19260
W20260929 20:34:45.491858 19322 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50397
W20260929 20:34:45.492825 29650 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 19260: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:34:45.493011 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 19260
2026-09-29T20:34:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2088 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:34:45 dist-test-slave-p29l krb5kdc[19359](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:34:45 dist-test-slave-p29l krb5kdc[19359](info): set up 2 sockets
Sep 29 20:34:45 dist-test-slave-p29l krb5kdc[19359](info): commencing operation
krb5kdc: starting...
W20260929 20:34:46.724917 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.196s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:34:46 dist-test-slave-p29l krb5kdc[19359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:34:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:34:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.244.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
I20260929 20:34:47.011026 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:44825
--webserver_interface=127.28.244.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:38209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:44825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:34:47.364347 19372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:34:47.364763 19372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:34:47.364951 19372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:34:47.372746 19372 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:34:47.372973 19372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:34:47.373086 19372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:34:47.373198 19372 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:34:47.373281 19372 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:34:47.383764 19372 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:38209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:44825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:44825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:34:47.385329 19372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:34:47.386773 19372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:34:47.399544 19377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:47.399506 19378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:47.399540 19380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:34:47.400701 19372 server_base.cc:1034] running on GCE node
I20260929 20:34:47.401598 19372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:34:47.403227 19372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:34:47.404477 19372 hybrid_clock.cc:648] HybridClock initialized: now 1790714087404419 us; error 48 us; skew 500 ppm
I20260929 20:34:47.404865 19372 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:34:47 dist-test-slave-p29l krb5kdc[19359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714087, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:34:47.408355 19372 init.cc:377] Logged in from keytab as kudu/127.28.244.190@KRBTEST.COM (short username kudu)
I20260929 20:34:47.410084 19372 webserver.cc:460] Webserver started at http://127.28.244.190:40895/ using document root <none> and password file <none>
I20260929 20:34:47.410845 19372 fs_manager.cc:359] Metadata directory not provided
I20260929 20:34:47.411005 19372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:34:47.411331 19372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:34:47.414589 19372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "4309906a98f54d2a821654c9a538efba"
format_stamp: "Formatted at 2026-09-29 20:34:47 on dist-test-slave-p29l"
server_key: "ce44abcc2b58208cb3e0452b1730904b"
server_key_iv: "d3bb75edf87497a4aa95a3fa9b23c848"
server_key_version: "encryptionkey@0"
I20260929 20:34:47.415436 19372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "4309906a98f54d2a821654c9a538efba"
format_stamp: "Formatted at 2026-09-29 20:34:47 on dist-test-slave-p29l"
server_key: "ce44abcc2b58208cb3e0452b1730904b"
server_key_iv: "d3bb75edf87497a4aa95a3fa9b23c848"
server_key_version: "encryptionkey@0"
I20260929 20:34:47.421368 19372 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 20:34:47.425657 19387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:47.427179 19372 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:34:47.427443 19372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "4309906a98f54d2a821654c9a538efba"
format_stamp: "Formatted at 2026-09-29 20:34:47 on dist-test-slave-p29l"
server_key: "ce44abcc2b58208cb3e0452b1730904b"
server_key_iv: "d3bb75edf87497a4aa95a3fa9b23c848"
server_key_version: "encryptionkey@0"
I20260929 20:34:47.427700 19372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:34:47.448985 19372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:34:47.453706 19372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:34:47.454047 19372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:34:47.474268 19372 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:44825
I20260929 20:34:47.474298 19438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:44825 every 8 connection(s)
I20260929 20:34:47.475822 19372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:34:47.478823 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 19372
I20260929 20:34:47.479030 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:34:47.479532 29650 external_mini_cluster.cc:1428] Setting key e46e81e601720aa699ca6f013d1aba61
I20260929 20:34:47.480163 19439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 20:34:47 dist-test-slave-p29l krb5kdc[19359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:34:47.495141 19439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4309906a98f54d2a821654c9a538efba: Bootstrap starting.
I20260929 20:34:47.499205 19439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4309906a98f54d2a821654c9a538efba: Neither blocks nor log segments found. Creating new log.
I20260929 20:34:47.500586 19439 log.cc:824] T 00000000000000000000000000000000 P 4309906a98f54d2a821654c9a538efba: Log is configured to *not* fsync() on all Append() calls
I20260929 20:34:47.504124 19439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4309906a98f54d2a821654c9a538efba: No bootstrap required, opened a new log
I20260929 20:34:47.505368 19442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:47.481558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54779 (local address 127.28.244.190:44825)
0929 20:34:47.482413 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:47.482427 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:47.482475 (+ 48us) server_negotiation.cc:407] Connection header received
0929 20:34:47.484345 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:47.484371 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:47.484868 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:47.485367 (+ 499us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:47.487091 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:47.488706 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:47.490215 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:47.491109 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:47.491360 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:47.496519 (+ 5159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:47.496529 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:47.496547 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:47.496601 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:47.500371 (+ 3770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:47.501357 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:47.501376 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:47.501388 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:47.501602 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:47.502076 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:47.502083 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:47.502087 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:47.502711 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:47.503056 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:47.503628 (+ 572us) server_negotiation.cc:299] Negotiation successful
0929 20:34:47.504179 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":377,"threads_started":1}
I20260929 20:34:47.509904 29650 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:34:47.513932 19439 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4309906a98f54d2a821654c9a538efba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4309906a98f54d2a821654c9a538efba" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 44825 } }
I20260929 20:34:47.514482 19439 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4309906a98f54d2a821654c9a538efba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:34:47.514621 19439 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4309906a98f54d2a821654c9a538efba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4309906a98f54d2a821654c9a538efba, State: Initialized, Role: FOLLOWER
I20260929 20:34:47.515564 19439 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4309906a98f54d2a821654c9a538efba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4309906a98f54d2a821654c9a538efba" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 44825 } }
I20260929 20:34:47.516040 19439 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4309906a98f54d2a821654c9a538efba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:34:47.516237 19439 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4309906a98f54d2a821654c9a538efba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:34:47.516486 19439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4309906a98f54d2a821654c9a538efba [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:34:47.519161 19439 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4309906a98f54d2a821654c9a538efba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4309906a98f54d2a821654c9a538efba" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 44825 } }
I20260929 20:34:47.520020 19439 leader_election.cc:302] T 00000000000000000000000000000000 P 4309906a98f54d2a821654c9a538efba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4309906a98f54d2a821654c9a538efba; no voters:
I20260929 20:34:47.520949 19439 leader_election.cc:288] T 00000000000000000000000000000000 P 4309906a98f54d2a821654c9a538efba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:34:47.521046 19449 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4309906a98f54d2a821654c9a538efba [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:34:47.524987 19439 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4309906a98f54d2a821654c9a538efba [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:34:47.526780 19449 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4309906a98f54d2a821654c9a538efba [term 1 LEADER]: Becoming Leader. State: Replica: 4309906a98f54d2a821654c9a538efba, State: Running, Role: LEADER
I20260929 20:34:47.527863 19449 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4309906a98f54d2a821654c9a538efba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4309906a98f54d2a821654c9a538efba" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 44825 } }
I20260929 20:34:47.532001 19451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4309906a98f54d2a821654c9a538efba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4309906a98f54d2a821654c9a538efba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4309906a98f54d2a821654c9a538efba" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 44825 } } }
I20260929 20:34:47.532581 19451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4309906a98f54d2a821654c9a538efba [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:47.533427 19442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:47.515140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52169 (local address 127.28.244.190:44825)
0929 20:34:47.515366 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:47.515372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:47.515952 (+ 580us) server_negotiation.cc:407] Connection header received
0929 20:34:47.516195 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:47.516201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:47.516284 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:47.516402 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:47.518092 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:47.518907 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:47.520525 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:47.521260 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:47.521430 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:47.524448 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:47.524461 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:47.524477 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:47.524561 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:47.528497 (+ 3936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:47.529500 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:47.529508 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:47.529517 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:47.529665 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:47.530142 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:47.530148 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:47.530154 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:47.530670 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:47.532223 (+ 1553us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:47.532457 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 20:34:47.532745 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:34:47.533762 19456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:34:47.533664 19450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4309906a98f54d2a821654c9a538efba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4309906a98f54d2a821654c9a538efba. Latest consensus state: current_term: 1 leader_uuid: "4309906a98f54d2a821654c9a538efba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4309906a98f54d2a821654c9a538efba" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 44825 } } }
I20260929 20:34:47.534163 19450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4309906a98f54d2a821654c9a538efba [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:47.541586 19456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:34:47.556216 19456 catalog_manager.cc:1347] Generated new cluster ID: d8e2fcf7419944f4995b3792751a74a0
I20260929 20:34:47.556358 19456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:34:47.565570 19456 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:34:47.567889 19456 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:34:47.586822 19456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4309906a98f54d2a821654c9a538efba: Generated new TSK 0
I20260929 20:34:47.588225 19456 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:34:47.604048 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 19372
I20260929 20:34:47.605443 19434 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:34:47.702702 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 19372
2026-09-29T20:34:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2212 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:34:47 dist-test-slave-p29l krb5kdc[19473](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:34:47 dist-test-slave-p29l krb5kdc[19473](info): set up 2 sockets
Sep 29 20:34:47 dist-test-slave-p29l krb5kdc[19473](info): commencing operation
krb5kdc: starting...
W20260929 20:34:48.813069 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:34:48 dist-test-slave-p29l krb5kdc[19473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:34:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:34:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.244.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
I20260929 20:34:49.092674 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:34971
--webserver_interface=127.28.244.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:46557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:34971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:34:49.442735 19486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:34:49.443071 19486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:34:49.443244 19486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:34:49.450848 19486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:34:49.450992 19486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:34:49.451088 19486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:34:49.451202 19486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:34:49.451267 19486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:34:49.461896 19486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:46557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:34971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:34971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:34:49.463402 19486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:34:49.464905 19486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:34:49.478121 19492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:49.478215 19494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:49.478242 19491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:34:49.479481 19486 server_base.cc:1034] running on GCE node
I20260929 20:34:49.480469 19486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:34:49.482103 19486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:34:49.483349 19486 hybrid_clock.cc:648] HybridClock initialized: now 1790714089483287 us; error 59 us; skew 500 ppm
I20260929 20:34:49.483729 19486 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:34:49 dist-test-slave-p29l krb5kdc[19473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714089, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:34:49.492051 19486 init.cc:377] Logged in from keytab as kudu/127.28.244.190@KRBTEST.COM (short username kudu)
I20260929 20:34:49.493821 19486 webserver.cc:460] Webserver started at http://127.28.244.190:34549/ using document root <none> and password file <none>
I20260929 20:34:49.494659 19486 fs_manager.cc:359] Metadata directory not provided
I20260929 20:34:49.494823 19486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:34:49.495157 19486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:34:49.498497 19486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "d919a2807a3c453183e910b8114fb43f"
format_stamp: "Formatted at 2026-09-29 20:34:49 on dist-test-slave-p29l"
server_key: "d51e65639bdcd9fcafb1a075de43963e"
server_key_iv: "496fb47ca7afa27478b65237a9ed6679"
server_key_version: "encryptionkey@0"
I20260929 20:34:49.499396 19486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "d919a2807a3c453183e910b8114fb43f"
format_stamp: "Formatted at 2026-09-29 20:34:49 on dist-test-slave-p29l"
server_key: "d51e65639bdcd9fcafb1a075de43963e"
server_key_iv: "496fb47ca7afa27478b65237a9ed6679"
server_key_version: "encryptionkey@0"
I20260929 20:34:49.505291 19486 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 20:34:49.509557 19501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:49.511102 19486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.004s
I20260929 20:34:49.511332 19486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "d919a2807a3c453183e910b8114fb43f"
format_stamp: "Formatted at 2026-09-29 20:34:49 on dist-test-slave-p29l"
server_key: "d51e65639bdcd9fcafb1a075de43963e"
server_key_iv: "496fb47ca7afa27478b65237a9ed6679"
server_key_version: "encryptionkey@0"
I20260929 20:34:49.511574 19486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:34:49.535739 19486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:34:49.540710 19486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:34:49.541046 19486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:34:49.561287 19486 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:34971
I20260929 20:34:49.561305 19552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:34971 every 8 connection(s)
I20260929 20:34:49.562836 19486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:34:49.567034 19553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:34:49.571031 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 19486
I20260929 20:34:49.571216 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:34:49.571762 29650 external_mini_cluster.cc:1428] Setting key ff344f49b1f6f3d6859b8a5ff469bc14
I20260929 20:34:49.581813 19553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d919a2807a3c453183e910b8114fb43f: Bootstrap starting.
I20260929 20:34:49.585500 19553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d919a2807a3c453183e910b8114fb43f: Neither blocks nor log segments found. Creating new log.
I20260929 20:34:49.586880 19553 log.cc:824] T 00000000000000000000000000000000 P d919a2807a3c453183e910b8114fb43f: Log is configured to *not* fsync() on all Append() calls
I20260929 20:34:49.587396 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:49.573952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59367 (local address 127.28.244.190:34971)
0929 20:34:49.574883 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:49.574896 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:49.574947 (+ 51us) server_negotiation.cc:407] Connection header received
0929 20:34:49.576957 (+ 2010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:49.577023 (+ 66us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:49.577768 (+ 745us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:49.578389 (+ 621us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:49.579049 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:49.580689 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:49.582259 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:49.583047 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:49.583282 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:49.584660 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:49.584669 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:49.584685 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:34:49.585343 (+ 658us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:49.585450 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:34:49.585480 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:49.585635 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:49.585899 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 20:34:49.586219 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":448,"threads_started":1}
I20260929 20:34:49.590284 19553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d919a2807a3c453183e910b8114fb43f: No bootstrap required, opened a new log
I20260929 20:34:49.591673 29650 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:34:49.600421 19553 raft_consensus.cc:348] T 00000000000000000000000000000000 P d919a2807a3c453183e910b8114fb43f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d919a2807a3c453183e910b8114fb43f" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34971 } }
I20260929 20:34:49.600906 19553 raft_consensus.cc:374] T 00000000000000000000000000000000 P d919a2807a3c453183e910b8114fb43f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:34:49.601045 19553 raft_consensus.cc:729] T 00000000000000000000000000000000 P d919a2807a3c453183e910b8114fb43f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d919a2807a3c453183e910b8114fb43f, State: Initialized, Role: FOLLOWER
I20260929 20:34:49.601862 19553 consensus_queue.cc:260] T 00000000000000000000000000000000 P d919a2807a3c453183e910b8114fb43f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d919a2807a3c453183e910b8114fb43f" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34971 } }
I20260929 20:34:49.602214 19553 raft_consensus.cc:388] T 00000000000000000000000000000000 P d919a2807a3c453183e910b8114fb43f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:34:49.602413 19553 raft_consensus.cc:482] T 00000000000000000000000000000000 P d919a2807a3c453183e910b8114fb43f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:34:49.602625 19553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d919a2807a3c453183e910b8114fb43f [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:34:49.604496 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:49.596616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60693 (local address 127.28.244.190:34971)
0929 20:34:49.596852 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:49.596861 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:49.597312 (+ 451us) server_negotiation.cc:407] Connection header received
0929 20:34:49.597507 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:49.597519 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:49.597609 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:49.597756 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:49.598319 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:49.599582 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:49.601293 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:49.602050 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:49.602215 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:49.603554 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:49.603563 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:49.603568 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:34:49.603593 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:49.603663 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:34:49.603687 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:49.603798 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:49.603994 (+ 196us) server_negotiation.cc:299] Negotiation successful
0929 20:34:49.604114 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 20:34:49.605901 19553 raft_consensus.cc:504] T 00000000000000000000000000000000 P d919a2807a3c453183e910b8114fb43f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d919a2807a3c453183e910b8114fb43f" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34971 } }
I20260929 20:34:49.606514 19553 leader_election.cc:302] T 00000000000000000000000000000000 P d919a2807a3c453183e910b8114fb43f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d919a2807a3c453183e910b8114fb43f; no voters:
I20260929 20:34:49.607160 19553 leader_election.cc:288] T 00000000000000000000000000000000 P d919a2807a3c453183e910b8114fb43f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:34:49.607309 19563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d919a2807a3c453183e910b8114fb43f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:34:49.608165 19563 raft_consensus.cc:686] T 00000000000000000000000000000000 P d919a2807a3c453183e910b8114fb43f [term 1 LEADER]: Becoming Leader. State: Replica: d919a2807a3c453183e910b8114fb43f, State: Running, Role: LEADER
I20260929 20:34:49.608749 19563 consensus_queue.cc:237] T 00000000000000000000000000000000 P d919a2807a3c453183e910b8114fb43f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d919a2807a3c453183e910b8114fb43f" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34971 } }
I20260929 20:34:49.609613 19553 sys_catalog.cc:564] T 00000000000000000000000000000000 P d919a2807a3c453183e910b8114fb43f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:34:49.611732 19564 sys_catalog.cc:455] T 00000000000000000000000000000000 P d919a2807a3c453183e910b8114fb43f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d919a2807a3c453183e910b8114fb43f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d919a2807a3c453183e910b8114fb43f" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34971 } } }
I20260929 20:34:49.612272 19564 sys_catalog.cc:458] T 00000000000000000000000000000000 P d919a2807a3c453183e910b8114fb43f [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:49.612848 19565 sys_catalog.cc:455] T 00000000000000000000000000000000 P d919a2807a3c453183e910b8114fb43f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d919a2807a3c453183e910b8114fb43f. Latest consensus state: current_term: 1 leader_uuid: "d919a2807a3c453183e910b8114fb43f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d919a2807a3c453183e910b8114fb43f" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 34971 } } }
I20260929 20:34:49.613319 19565 sys_catalog.cc:458] T 00000000000000000000000000000000 P d919a2807a3c453183e910b8114fb43f [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:49.613868 19569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:34:49.622526 19569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:34:49.633057 19569 catalog_manager.cc:1347] Generated new cluster ID: 27dc807506454ce98a52e50fe475c60e
I20260929 20:34:49.633208 19569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:34:49.669070 19569 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:34:49.671213 19569 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:34:49.680299 19569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d919a2807a3c453183e910b8114fb43f: Generated new TSK 0
I20260929 20:34:49.681450 19569 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:34:49.693065 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 19486
W20260929 20:34:49.694388 19548 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59367
W20260929 20:34:49.695179 29650 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 19486: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:34:49.695346 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 19486
2026-09-29T20:34:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1991 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:34:49 dist-test-slave-p29l krb5kdc[19584](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:34:49 dist-test-slave-p29l krb5kdc[19584](info): set up 2 sockets
Sep 29 20:34:49 dist-test-slave-p29l krb5kdc[19584](info): commencing operation
krb5kdc: starting...
W20260929 20:34:50.817059 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.006s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:34:50 dist-test-slave-p29l krb5kdc[19584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:34:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:34:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.244.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
I20260929 20:34:51.088181 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:37959
--webserver_interface=127.28.244.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:41437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:37959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:34:51.428627 19597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:34:51.428993 19597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:34:51.429164 19597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:34:51.436530 19597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:34:51.436695 19597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:34:51.436795 19597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:34:51.436920 19597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:34:51.437000 19597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:34:51.447175 19597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:41437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:37959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:37959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:34:51.448593 19597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:34:51.449946 19597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:34:51.462285 19603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:51.462312 19602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:51.462507 19605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:34:51.463344 19597 server_base.cc:1034] running on GCE node
I20260929 20:34:51.464169 19597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:34:51.465600 19597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:34:51.466815 19597 hybrid_clock.cc:648] HybridClock initialized: now 1790714091466758 us; error 51 us; skew 500 ppm
I20260929 20:34:51.467199 19597 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:34:51 dist-test-slave-p29l krb5kdc[19584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714091, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:34:51.470374 19597 init.cc:377] Logged in from keytab as kudu/127.28.244.190@KRBTEST.COM (short username kudu)
I20260929 20:34:51.471979 19597 webserver.cc:460] Webserver started at http://127.28.244.190:42297/ using document root <none> and password file <none>
I20260929 20:34:51.472682 19597 fs_manager.cc:359] Metadata directory not provided
I20260929 20:34:51.472816 19597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:34:51.473150 19597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:34:51.476276 19597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "1ac6753576f349b2a693e5e09157185e"
format_stamp: "Formatted at 2026-09-29 20:34:51 on dist-test-slave-p29l"
server_key: "bcccd093adbbaae05c786e6a3d4b9582"
server_key_iv: "843ffbb425426bf6547f99d88196ea1b"
server_key_version: "encryptionkey@0"
I20260929 20:34:51.477061 19597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "1ac6753576f349b2a693e5e09157185e"
format_stamp: "Formatted at 2026-09-29 20:34:51 on dist-test-slave-p29l"
server_key: "bcccd093adbbaae05c786e6a3d4b9582"
server_key_iv: "843ffbb425426bf6547f99d88196ea1b"
server_key_version: "encryptionkey@0"
I20260929 20:34:51.482471 19597 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 20:34:51.486456 19613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:51.488071 19597 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:34:51.488271 19597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "1ac6753576f349b2a693e5e09157185e"
format_stamp: "Formatted at 2026-09-29 20:34:51 on dist-test-slave-p29l"
server_key: "bcccd093adbbaae05c786e6a3d4b9582"
server_key_iv: "843ffbb425426bf6547f99d88196ea1b"
server_key_version: "encryptionkey@0"
I20260929 20:34:51.488514 19597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:34:51.495011 19597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:34:51.499140 19597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:34:51.499459 19597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:34:51.518606 19597 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:37959
I20260929 20:34:51.518644 19664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:37959 every 8 connection(s)
I20260929 20:34:51.520149 19597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:34:51.524072 19665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:34:51.524600 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 19597
I20260929 20:34:51.524781 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:34:51.525308 29650 external_mini_cluster.cc:1428] Setting key 96e6fab9879180ca765244401761bfa8
I20260929 20:34:51.533959 19668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:51.527189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48377 (local address 127.28.244.190:37959)
0929 20:34:51.528143 (+ 954us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:51.528154 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:51.528193 (+ 39us) server_negotiation.cc:407] Connection header received
0929 20:34:51.530451 (+ 2258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:51.530475 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:51.530835 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:51.531218 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:51.532382 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:51.532389 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:51.532401 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:34:51.532418 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:51.532483 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:34:51.532509 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:51.532697 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:51.532801 (+ 104us) server_negotiation.cc:299] Negotiation successful
0929 20:34:51.533076 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":738,"thread_start_us":428,"threads_started":1}
I20260929 20:34:51.537664 29650 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:34:51.537897 19665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ac6753576f349b2a693e5e09157185e: Bootstrap starting.
I20260929 20:34:51.541867 19665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ac6753576f349b2a693e5e09157185e: Neither blocks nor log segments found. Creating new log.
I20260929 20:34:51.543219 19665 log.cc:824] T 00000000000000000000000000000000 P 1ac6753576f349b2a693e5e09157185e: Log is configured to *not* fsync() on all Append() calls
I20260929 20:34:51.546030 19668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:51.543034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54803 (local address 127.28.244.190:37959)
0929 20:34:51.543238 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:51.543245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:51.543704 (+ 459us) server_negotiation.cc:407] Connection header received
0929 20:34:51.543921 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:51.543927 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:51.544007 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:51.544094 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:51.545284 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:51.545289 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:51.545294 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:34:51.545312 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:51.545356 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:34:51.545376 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:51.545498 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:51.545652 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 20:34:51.545717 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:34:51.546808 19665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ac6753576f349b2a693e5e09157185e: No bootstrap required, opened a new log
I20260929 20:34:51.554436 19665 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ac6753576f349b2a693e5e09157185e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ac6753576f349b2a693e5e09157185e" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 37959 } }
I20260929 20:34:51.554857 19665 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ac6753576f349b2a693e5e09157185e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:34:51.554978 19665 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ac6753576f349b2a693e5e09157185e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ac6753576f349b2a693e5e09157185e, State: Initialized, Role: FOLLOWER
I20260929 20:34:51.555583 19665 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ac6753576f349b2a693e5e09157185e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ac6753576f349b2a693e5e09157185e" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 37959 } }
I20260929 20:34:51.555840 19665 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ac6753576f349b2a693e5e09157185e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:34:51.556051 19665 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ac6753576f349b2a693e5e09157185e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:34:51.556245 19665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ac6753576f349b2a693e5e09157185e [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:34:51.558780 19665 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ac6753576f349b2a693e5e09157185e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ac6753576f349b2a693e5e09157185e" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 37959 } }
I20260929 20:34:51.559320 19665 leader_election.cc:302] T 00000000000000000000000000000000 P 1ac6753576f349b2a693e5e09157185e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ac6753576f349b2a693e5e09157185e; no voters:
I20260929 20:34:51.560075 19665 leader_election.cc:288] T 00000000000000000000000000000000 P 1ac6753576f349b2a693e5e09157185e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:34:51.560262 19675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ac6753576f349b2a693e5e09157185e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:34:51.561082 19675 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ac6753576f349b2a693e5e09157185e [term 1 LEADER]: Becoming Leader. State: Replica: 1ac6753576f349b2a693e5e09157185e, State: Running, Role: LEADER
I20260929 20:34:51.561734 19675 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ac6753576f349b2a693e5e09157185e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ac6753576f349b2a693e5e09157185e" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 37959 } }
I20260929 20:34:51.562426 19665 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ac6753576f349b2a693e5e09157185e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:34:51.565215 19676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ac6753576f349b2a693e5e09157185e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ac6753576f349b2a693e5e09157185e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ac6753576f349b2a693e5e09157185e" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 37959 } } }
I20260929 20:34:51.565259 19677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ac6753576f349b2a693e5e09157185e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ac6753576f349b2a693e5e09157185e. Latest consensus state: current_term: 1 leader_uuid: "1ac6753576f349b2a693e5e09157185e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ac6753576f349b2a693e5e09157185e" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 37959 } } }
I20260929 20:34:51.565773 19676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ac6753576f349b2a693e5e09157185e [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:51.565799 19677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ac6753576f349b2a693e5e09157185e [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:51.568545 19682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:34:51.574527 19682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:34:51.584270 19682 catalog_manager.cc:1347] Generated new cluster ID: e76455da43a746eeac1232a873bd55e3
I20260929 20:34:51.584408 19682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:34:51.598899 19682 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:34:51.599344 19682 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:34:51.608430 19682 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ac6753576f349b2a693e5e09157185e: Generated new TSK 0
I20260929 20:34:51.609331 19682 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:34:51.627584 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 19597
W20260929 20:34:51.629333 19660 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48377
W20260929 20:34:51.630134 29650 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 19597: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:34:51.630291 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 19597
2026-09-29T20:34:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1934 ms)
[----------] 18 tests from AuthTokenIssuingTest (36815 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:34:51 dist-test-slave-p29l krb5kdc[19696](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:34:51 dist-test-slave-p29l krb5kdc[19696](info): set up 2 sockets
Sep 29 20:34:51 dist-test-slave-p29l krb5kdc[19696](info): commencing operation
krb5kdc: starting...
W20260929 20:34:52.748006 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:34:52 dist-test-slave-p29l krb5kdc[19696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:34:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:34:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.244.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
I20260929 20:34:53.025287 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:42537
--webserver_interface=127.28.244.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:42537,127.28.244.189:36903,127.28.244.188:36063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:34:53.365952 19709 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 20:34:53.366358 19709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:34:53.366499 19709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:34:53.366609 19709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:34:53.374321 19709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:34:53.374437 19709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:34:53.374534 19709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:34:53.374636 19709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:34:53.374723 19709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:34:53.385142 19709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:42537,127.28.244.189:36903,127.28.244.188:36063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:42537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:34:53.386545 19709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:34:53.387887 19709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:34:53.400117 19714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:53.400158 19715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:53.400184 19717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:34:53.405547 19709 server_base.cc:1034] running on GCE node
I20260929 20:34:53.406502 19709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:34:53.408197 19709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:34:53.409479 19709 hybrid_clock.cc:648] HybridClock initialized: now 1790714093409426 us; error 48 us; skew 500 ppm
I20260929 20:34:53.409865 19709 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:34:53 dist-test-slave-p29l krb5kdc[19696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714093, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:34:53.413383 19709 init.cc:377] Logged in from keytab as kudu/127.28.244.190@KRBTEST.COM (short username kudu)
I20260929 20:34:53.415150 19709 webserver.cc:460] Webserver started at http://127.28.244.190:39161/ using document root <none> and password file <none>
I20260929 20:34:53.416121 19709 fs_manager.cc:359] Metadata directory not provided
I20260929 20:34:53.416270 19709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:34:53.416635 19709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:34:53.420113 19709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "56c07fd2e1fc4521977b3f9b6e5a4c8f"
format_stamp: "Formatted at 2026-09-29 20:34:53 on dist-test-slave-p29l"
server_key: "a629adda328b1e3f3fb1638717ba74db"
server_key_iv: "9f8a3b8548841a53a4a5afa36f347baa"
server_key_version: "encryptionkey@0"
I20260929 20:34:53.421051 19709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "56c07fd2e1fc4521977b3f9b6e5a4c8f"
format_stamp: "Formatted at 2026-09-29 20:34:53 on dist-test-slave-p29l"
server_key: "a629adda328b1e3f3fb1638717ba74db"
server_key_iv: "9f8a3b8548841a53a4a5afa36f347baa"
server_key_version: "encryptionkey@0"
I20260929 20:34:53.427158 19709 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260929 20:34:53.430964 19724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:53.432474 19709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:34:53.432677 19709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "56c07fd2e1fc4521977b3f9b6e5a4c8f"
format_stamp: "Formatted at 2026-09-29 20:34:53 on dist-test-slave-p29l"
server_key: "a629adda328b1e3f3fb1638717ba74db"
server_key_iv: "9f8a3b8548841a53a4a5afa36f347baa"
server_key_version: "encryptionkey@0"
I20260929 20:34:53.432938 19709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:34:53.453125 19709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:34:53.457734 19709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:34:53.458038 19709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:34:53.478739 19709 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:42537
I20260929 20:34:53.478746 19776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:42537 every 8 connection(s)
I20260929 20:34:53.480343 19709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:34:53.482450 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 19709
I20260929 20:34:53.482679 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:34:53.483511 29650 external_mini_cluster.cc:1428] Setting key 8c0387f018a13415159b49ad3d905ef1
I20260929 20:34:53.485368 19777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:34:53.495496 19777 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.244.190" port: 42537 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:34:53 dist-test-slave-p29l krb5kdc[19696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714093, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.244.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.189@KRBTEST.COM" created.
I20260929 20:34:53.519639 19781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:53.498853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55462 (local address 127.28.244.190:42537)
0929 20:34:53.500557 (+ 1704us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:53.500567 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:53.500622 (+ 55us) server_negotiation.cc:407] Connection header received
0929 20:34:53.502231 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:53.502241 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:53.502551 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:53.502772 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:53.503971 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:53.505888 (+ 1917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:53.507308 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:53.508357 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:53.508561 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:53.511943 (+ 3382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:53.511961 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:53.511982 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:53.512027 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:53.515142 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:53.515909 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:53.515922 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:53.515924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:53.516078 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:53.516466 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:53.516477 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:53.516480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:53.517010 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:53.517224 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:53.517777 (+ 553us) server_negotiation.cc:299] Negotiation successful
0929 20:34:53.518130 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":1477,"spinlock_wait_cycles":27008,"thread_start_us":1272,"threads_started":1}
I20260929 20:34:53.519675 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:53.498854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:42537 (local address 127.0.0.1:55462)
0929 20:34:53.499876 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:53.499939 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:53.501833 (+ 1894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:53.502790 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:53.502795 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:53.503380 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:34:53.503727 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:53.503745 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:53.506081 (+ 2336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:53.506085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:53.507162 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:53.507174 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:53.508573 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:53.508578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:53.508713 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:53.509423 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:34:53.509447 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:34:53.511641 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:34:53.515294 (+ 3653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:53.515305 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:53.515319 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:53.515731 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:53.516193 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:53.516199 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:53.516202 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:53.516352 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:53.517153 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:34:53.517164 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:34:53.517393 (+ 229us) client_negotiation.cc:769] Sending connection context
0929 20:34:53.517642 (+ 249us) client_negotiation.cc:240] Negotiation successful
0929 20:34:53.518250 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":135,"thread_start_us":390,"threads_started":1}
I20260929 20:34:53.522921 19777 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.244.189" port: 36903 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:34:53.524070 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:53.523624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:36903 (local address 127.0.0.1:42844)
0929 20:34:53.523789 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:53.523947 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.189:36903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 20:34:53.524302 19728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.244.189:36903: connect: Connection refused (error 111)
I20260929 20:34:53.525911 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:53.525552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:36903 (local address 127.0.0.1:42860)
0929 20:34:53.525721 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:53.525787 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.189:36903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 20:34:53.526095 19777 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.189:36903: Network error: Client connection negotiation failed: client connection to 127.28.244.189:36903: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.189.keytab.
Entry for principal kudu/127.28.244.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.189.keytab.
I20260929 20:34:53.547605 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.189:36903
--webserver_interface=127.28.244.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.189.keytab
--principal=kudu/127.28.244.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:42537,127.28.244.189:36903,127.28.244.188:36063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:34:53.575452 19777 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.189" port: 36903 } attempt: 1
I20260929 20:34:53.576743 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:53.576286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:36903 (local address 127.0.0.1:42874)
0929 20:34:53.576491 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:53.576611 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.189:36903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 20:34:53.577885 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:53.577539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:36903 (local address 127.0.0.1:42880)
0929 20:34:53.577689 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:53.577759 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.189:36903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 20:34:53.578150 19777 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.189:36903: Network error: Client connection negotiation failed: client connection to 127.28.244.189:36903: connect: Connection refused (error 111)
I20260929 20:34:53.646449 19777 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.189" port: 36903 } attempt: 2
I20260929 20:34:53.647662 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:53.647175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:36903 (local address 127.0.0.1:42884)
0929 20:34:53.647430 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:53.647522 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.189:36903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 20:34:53.648737 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:53.648375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:36903 (local address 127.0.0.1:42896)
0929 20:34:53.648525 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:53.648613 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.189:36903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 20:34:53.648926 19777 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.189:36903: Network error: Client connection negotiation failed: client connection to 127.28.244.189:36903: connect: Connection refused (error 111)
I20260929 20:34:53.740171 19777 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.189" port: 36903 } attempt: 3
I20260929 20:34:53.741326 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:53.740891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:36903 (local address 127.0.0.1:42902)
0929 20:34:53.741106 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:53.741194 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.189:36903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 20:34:53.742372 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:53.742043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:36903 (local address 127.0.0.1:42906)
0929 20:34:53.742168 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:53.742233 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.189:36903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 20:34:53.742636 19777 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.189:36903: Network error: Client connection negotiation failed: client connection to 127.28.244.189:36903: connect: Connection refused (error 111)
I20260929 20:34:53.885908 19777 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.189" port: 36903 } attempt: 4
I20260929 20:34:53.887226 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:53.886654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:36903 (local address 127.0.0.1:42922)
0929 20:34:53.886973 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:53.887075 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.189:36903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
I20260929 20:34:53.888347 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:53.887956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:36903 (local address 127.0.0.1:42932)
0929 20:34:53.888138 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:53.888224 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.189:36903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260929 20:34:53.888589 19777 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.189:36903: Network error: Client connection negotiation failed: client connection to 127.28.244.189:36903: connect: Connection refused (error 111)
W20260929 20:34:53.889245 19784 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 20:34:53.889611 19784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:34:53.889694 19784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:34:53.889864 19784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:34:53.897344 19784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:34:53.897436 19784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:34:53.897532 19784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:34:53.897619 19784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:34:53.897727 19784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:34:53.908710 19784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:42537,127.28.244.189:36903,127.28.244.188:36063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.189.keytab
--principal=kudu/127.28.244.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.189:36903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.244.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:34:53.910043 19784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:34:53.911351 19784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:34:53.923583 19789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:53.923623 19792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:53.923578 19790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:34:53.925086 19784 server_base.cc:1034] running on GCE node
I20260929 20:34:53.925843 19784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:34:53.927239 19784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:34:53.928483 19784 hybrid_clock.cc:648] HybridClock initialized: now 1790714093928423 us; error 27 us; skew 500 ppm
I20260929 20:34:53.928916 19784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:34:53 dist-test-slave-p29l krb5kdc[19696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714093, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:34:53.932204 19784 init.cc:377] Logged in from keytab as kudu/127.28.244.189@KRBTEST.COM (short username kudu)
I20260929 20:34:53.933694 19784 webserver.cc:460] Webserver started at http://127.28.244.189:40315/ using document root <none> and password file <none>
I20260929 20:34:53.934366 19784 fs_manager.cc:359] Metadata directory not provided
I20260929 20:34:53.934451 19784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:34:53.934710 19784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:34:53.937739 19784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-1/data/instance:
uuid: "642de662dfad45a28e4afc8712f027c2"
format_stamp: "Formatted at 2026-09-29 20:34:53 on dist-test-slave-p29l"
server_key: "f2977ae07c100909e76d0369a411c755"
server_key_iv: "d3edc5ac6dbd9c24dbbdc938c4d3f281"
server_key_version: "encryptionkey@0"
I20260929 20:34:53.938519 19784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-1/wal/instance:
uuid: "642de662dfad45a28e4afc8712f027c2"
format_stamp: "Formatted at 2026-09-29 20:34:53 on dist-test-slave-p29l"
server_key: "f2977ae07c100909e76d0369a411c755"
server_key_iv: "d3edc5ac6dbd9c24dbbdc938c4d3f281"
server_key_version: "encryptionkey@0"
I20260929 20:34:53.943761 19784 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 20:34:53.947240 19799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:53.948623 19784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:34:53.948804 19784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-1/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-1/wal
uuid: "642de662dfad45a28e4afc8712f027c2"
format_stamp: "Formatted at 2026-09-29 20:34:53 on dist-test-slave-p29l"
server_key: "f2977ae07c100909e76d0369a411c755"
server_key_iv: "d3edc5ac6dbd9c24dbbdc938c4d3f281"
server_key_version: "encryptionkey@0"
I20260929 20:34:53.949028 19784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:34:53.969805 19784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:34:53.974350 19784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:34:53.974632 19784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:34:53.993587 19784 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.189:36903
I20260929 20:34:53.993793 19850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.189:36903 every 8 connection(s)
I20260929 20:34:53.994987 19784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-1/data/info.pb
I20260929 20:34:53.998682 19851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:34:54.004245 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 19784
I20260929 20:34:54.004431 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-1/wal/instance
I20260929 20:34:54.004968 29650 external_mini_cluster.cc:1428] Setting key d8bd50ca563a2323cd4729438e3bed7f
I20260929 20:34:54.005661 19851 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.244.190" port: 42537 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:34:54 dist-test-slave-p29l krb5kdc[19696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714093, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.189@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:34:54.030900 19781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:54.009243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55476 (local address 127.28.244.190:42537)
0929 20:34:54.009467 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:54.009473 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:54.010701 (+ 1228us) server_negotiation.cc:407] Connection header received
0929 20:34:54.013695 (+ 2994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:54.013700 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:54.013774 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:54.013864 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:54.015907 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.017057 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.018560 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.019330 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.019472 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:54.023188 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:54.023198 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:54.023206 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:54.023255 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:54.027103 (+ 3848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:54.027954 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:54.027964 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:54.027970 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:54.028129 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:54.028616 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:54.028624 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:54.028629 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:54.029051 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:54.029218 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:54.030149 (+ 931us) server_negotiation.cc:299] Negotiation successful
0929 20:34:54.030441 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 20:34:54.031548 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:54.009669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:42537 (local address 127.0.0.1:55476)
0929 20:34:54.010509 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:54.010557 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:54.013368 (+ 2811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:54.014147 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:54.014160 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:54.014954 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:34:54.015665 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:54.015684 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.017225 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.017228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:54.018403 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:54.018415 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.019551 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.019556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:54.019758 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:54.020516 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:34:54.020543 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:34:54.022941 (+ 2398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:34:54.027272 (+ 4331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:54.027279 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:54.027295 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:54.027773 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:54.028270 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:54.028274 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:54.028277 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:54.028475 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:54.029234 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:34:54.029243 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:34:54.029677 (+ 434us) client_negotiation.cc:769] Sending connection context
0929 20:34:54.029964 (+ 287us) client_negotiation.cc:240] Negotiation successful
0929 20:34:54.030376 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":330,"threads_started":1}
WARNING: no policy specified for kudu/127.28.244.188@KRBTEST.COM; defaulting to no policy
I20260929 20:34:54.033391 19851 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.244.189" port: 36903 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.188@KRBTEST.COM" created.
Sep 29 20:34:54 dist-test-slave-p29l krb5kdc[19696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714093, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.189@KRBTEST.COM for kudu/127.28.244.189@KRBTEST.COM
I20260929 20:34:54.051430 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:54.034482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42946 (local address 127.28.244.189:36903)
0929 20:34:54.035056 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:54.035062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:54.035086 (+ 24us) server_negotiation.cc:407] Connection header received
0929 20:34:54.035128 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:54.035134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:54.035308 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:54.035415 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:54.036608 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.037818 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.039920 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.040527 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.040637 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:54.044722 (+ 4085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:54.044742 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:54.044766 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:54.044818 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:54.048236 (+ 3418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:54.049122 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:54.049129 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:54.049132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:54.049274 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:54.049747 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:54.049751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:54.049753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:54.050150 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:54.050329 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:54.050753 (+ 424us) server_negotiation.cc:299] Negotiation successful
0929 20:34:54.051123 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":159,"server-negotiator.queue_time_us":402,"thread_start_us":262,"threads_started":1}
I20260929 20:34:54.051455 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:54.034177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:36903 (local address 127.0.0.1:42946)
0929 20:34:54.034355 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:54.034374 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:54.034472 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:54.035468 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:54.035473 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:54.036160 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:34:54.036398 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:54.036414 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.037967 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.037976 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:54.039648 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:54.039668 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.040708 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.040717 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:54.040987 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:54.041666 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:34:54.041696 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:34:54.044477 (+ 2781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:34:54.048383 (+ 3906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:54.048391 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:54.048399 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:54.048985 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:54.049401 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:54.049409 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:54.049414 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:54.049612 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:54.050293 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:34:54.050301 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:34:54.050483 (+ 182us) client_negotiation.cc:769] Sending connection context
0929 20:34:54.050638 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 20:34:54.050982 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 20:34:54.054515 19851 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.244.188" port: 36063 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:34:54.055747 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:54.055184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:36063 (local address 127.0.0.1:36960)
0929 20:34:54.055383 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:54.055535 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:36063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 20:34:54.056073 19800 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.244.188:36063: connect: Connection refused (error 111)
I20260929 20:34:54.057546 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:54.057229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:36063 (local address 127.0.0.1:36962)
0929 20:34:54.057376 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:54.057445 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:36063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 20:34:54.057840 19851 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.188:36063: Network error: Client connection negotiation failed: client connection to 127.28.244.188:36063: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.188.keytab.
Entry for principal kudu/127.28.244.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.188.keytab.
I20260929 20:34:54.074575 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.188:36063
--webserver_interface=127.28.244.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.188.keytab
--principal=kudu/127.28.244.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:42537,127.28.244.189:36903,127.28.244.188:36063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:34:54.107209 19851 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.188" port: 36063 } attempt: 1
I20260929 20:34:54.108497 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:54.108000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:36063 (local address 127.0.0.1:36970)
0929 20:34:54.108232 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:54.108338 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:36063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260929 20:34:54.109493 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:54.109139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:36063 (local address 127.0.0.1:36978)
0929 20:34:54.109316 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:54.109390 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:36063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 20:34:54.109704 19851 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.188:36063: Network error: Client connection negotiation failed: client connection to 127.28.244.188:36063: connect: Connection refused (error 111)
I20260929 20:34:54.177852 19851 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.188" port: 36063 } attempt: 2
I20260929 20:34:54.179076 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:54.178601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:36063 (local address 127.0.0.1:36984)
0929 20:34:54.178812 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:54.178910 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:36063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 20:34:54.180146 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:54.179725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:36063 (local address 127.0.0.1:37000)
0929 20:34:54.179896 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:54.180000 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:36063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 20:34:54.180452 19851 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.188:36063: Network error: Client connection negotiation failed: client connection to 127.28.244.188:36063: connect: Connection refused (error 111)
I20260929 20:34:54.187899 19777 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.189" port: 36903 } attempt: 5
Sep 29 20:34:54 dist-test-slave-p29l krb5kdc[19696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714093, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for kudu/127.28.244.189@KRBTEST.COM
I20260929 20:34:54.201191 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:54.188631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:36903 (local address 127.0.0.1:42958)
0929 20:34:54.188843 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:54.188864 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:54.189040 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:54.189408 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:54.189413 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:54.189817 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:34:54.190000 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:54.190010 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.191085 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.191091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:54.192005 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:54.192017 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.192722 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.192728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:54.192964 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:54.193598 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:34:54.193615 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:34:54.195859 (+ 2244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:34:54.198774 (+ 2915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:54.198780 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:54.198784 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:54.199160 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:54.199533 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:54.199539 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:54.199542 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:54.199681 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:54.200237 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:34:54.200242 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:34:54.200399 (+ 157us) client_negotiation.cc:769] Sending connection context
0929 20:34:54.200542 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 20:34:54.200826 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 20:34:54.201187 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:54.188704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42958 (local address 127.28.244.189:36903)
0929 20:34:54.188879 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:54.188887 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:54.188953 (+ 66us) server_negotiation.cc:407] Connection header received
0929 20:34:54.189178 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:54.189184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:54.189272 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:54.189375 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:54.190145 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.190955 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.192140 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.192567 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.192695 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:54.196051 (+ 3356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:54.196056 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:54.196060 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:54.196091 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:54.198636 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:54.199277 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:54.199284 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:54.199287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:54.199416 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:54.199795 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:54.199800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:54.199830 (+ 30us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:54.200109 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:54.200259 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:54.200661 (+ 402us) server_negotiation.cc:299] Negotiation successful
0929 20:34:54.200865 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 20:34:54.202608 19777 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.244.188" port: 36063 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:34:54.203630 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:54.203277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:36063 (local address 127.0.0.1:37016)
0929 20:34:54.203429 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:54.203505 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:36063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 20:34:54.204576 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:54.204259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:36063 (local address 127.0.0.1:37028)
0929 20:34:54.204406 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:54.204474 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:36063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 20:34:54.204764 19777 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.188:36063: Network error: Client connection negotiation failed: client connection to 127.28.244.188:36063: connect: Connection refused (error 111)
I20260929 20:34:54.256012 19777 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.188" port: 36063 } attempt: 1
I20260929 20:34:54.257118 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:54.256652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:36063 (local address 127.0.0.1:37034)
0929 20:34:54.256850 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:54.256946 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:36063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 20:34:54.258065 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:54.257743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:36063 (local address 127.0.0.1:37048)
0929 20:34:54.257888 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:54.257960 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:36063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 20:34:54.258256 19777 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.188:36063: Network error: Client connection negotiation failed: client connection to 127.28.244.188:36063: connect: Connection refused (error 111)
I20260929 20:34:54.271685 19851 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.188" port: 36063 } attempt: 3
I20260929 20:34:54.272946 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:54.272480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:36063 (local address 127.0.0.1:37058)
0929 20:34:54.272671 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:54.272761 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:36063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 20:34:54.274051 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:54.273622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:36063 (local address 127.0.0.1:37070)
0929 20:34:54.273839 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:54.273915 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:36063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
W20260929 20:34:54.274303 19851 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.188:36063: Network error: Client connection negotiation failed: client connection to 127.28.244.188:36063: connect: Connection refused (error 111)
I20260929 20:34:54.326512 19777 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.188" port: 36063 } attempt: 2
I20260929 20:34:54.327741 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:54.327259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:36063 (local address 127.0.0.1:37074)
0929 20:34:54.327479 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:54.327585 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:36063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 20:34:54.328869 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:54.328562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:36063 (local address 127.0.0.1:37086)
0929 20:34:54.328697 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:54.328775 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:36063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 20:34:54.329079 19777 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.188:36063: Network error: Client connection negotiation failed: client connection to 127.28.244.188:36063: connect: Connection refused (error 111)
I20260929 20:34:54.417498 19851 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.188" port: 36063 } attempt: 4
I20260929 20:34:54.418761 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:54.418202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:36063 (local address 127.0.0.1:37088)
0929 20:34:54.418426 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:54.418535 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:36063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 20:34:54.419773 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:54.419429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:36063 (local address 127.0.0.1:37094)
0929 20:34:54.419586 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:54.419663 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:36063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 20:34:54.420090 19851 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.188:36063: Network error: Client connection negotiation failed: client connection to 127.28.244.188:36063: connect: Connection refused (error 111)
W20260929 20:34:54.424935 19858 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 20:34:54.425308 19858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:34:54.425432 19858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:34:54.425585 19858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:34:54.433054 19858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:34:54.433174 19858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:34:54.433295 19858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:34:54.433377 19858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:34:54.433491 19858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:34:54.435297 19777 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.188" port: 36063 } attempt: 3
I20260929 20:34:54.436445 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:54.435973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:36063 (local address 127.0.0.1:37106)
0929 20:34:54.436213 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:54.436297 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:36063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 20:34:54.437536 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:54.437169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:36063 (local address 127.0.0.1:37114)
0929 20:34:54.437359 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:54.437438 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:36063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260929 20:34:54.437758 19777 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.188:36063: Network error: Client connection negotiation failed: client connection to 127.28.244.188:36063: connect: Connection refused (error 111)
I20260929 20:34:54.444110 19858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:42537,127.28.244.189:36903,127.28.244.188:36063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.188.keytab
--principal=kudu/127.28.244.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.188:36063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.244.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:34:54.445449 19858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:34:54.446799 19858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:34:54.458846 19864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:54.458925 19866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:54.458902 19863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:34:54.465150 19858 server_base.cc:1034] running on GCE node
I20260929 20:34:54.466030 19858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:34:54.467785 19858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:34:54.469120 19858 hybrid_clock.cc:648] HybridClock initialized: now 1790714094469033 us; error 62 us; skew 500 ppm
I20260929 20:34:54.469496 19858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:34:54 dist-test-slave-p29l krb5kdc[19696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714094, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:34:54.472713 19858 init.cc:377] Logged in from keytab as kudu/127.28.244.188@KRBTEST.COM (short username kudu)
I20260929 20:34:54.474304 19858 webserver.cc:460] Webserver started at http://127.28.244.188:40069/ using document root <none> and password file <none>
I20260929 20:34:54.475024 19858 fs_manager.cc:359] Metadata directory not provided
I20260929 20:34:54.475117 19858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:34:54.475457 19858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:34:54.478793 19858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-2/data/instance:
uuid: "00926a6f776c4e50b11cec5b7a4f88d6"
format_stamp: "Formatted at 2026-09-29 20:34:54 on dist-test-slave-p29l"
server_key: "e28d4aee6c64a397bdef0a5c6caf004c"
server_key_iv: "dbccf39e0d3f606d40215d7c11d446a2"
server_key_version: "encryptionkey@0"
I20260929 20:34:54.479919 19858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-2/wal/instance:
uuid: "00926a6f776c4e50b11cec5b7a4f88d6"
format_stamp: "Formatted at 2026-09-29 20:34:54 on dist-test-slave-p29l"
server_key: "e28d4aee6c64a397bdef0a5c6caf004c"
server_key_iv: "dbccf39e0d3f606d40215d7c11d446a2"
server_key_version: "encryptionkey@0"
I20260929 20:34:54.486339 19858 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 20:34:54.490737 19873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:54.492225 19858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:34:54.492409 19858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-2/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-2/wal
uuid: "00926a6f776c4e50b11cec5b7a4f88d6"
format_stamp: "Formatted at 2026-09-29 20:34:54 on dist-test-slave-p29l"
server_key: "e28d4aee6c64a397bdef0a5c6caf004c"
server_key_iv: "dbccf39e0d3f606d40215d7c11d446a2"
server_key_version: "encryptionkey@0"
I20260929 20:34:54.492650 19858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:34:54.508445 19858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:34:54.512502 19858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:34:54.512784 19858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:34:54.531337 19924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.188:36063 every 8 connection(s)
I20260929 20:34:54.531373 19858 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.188:36063
I20260929 20:34:54.532902 19858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-2/data/info.pb
I20260929 20:34:54.536751 19925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:34:54.540930 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 19858
I20260929 20:34:54.541126 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713721201454-29650-0/minicluster-data/master-2/wal/instance
I20260929 20:34:54.541646 29650 external_mini_cluster.cc:1428] Setting key c8a760c4464e89bd97c5207646852a66
I20260929 20:34:54.544349 19925 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.244.190" port: 42537 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:34:54 dist-test-slave-p29l krb5kdc[19696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.188@KRBTEST.COM
Sep 29 20:34:54 dist-test-slave-p29l krb5kdc[19696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714094, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.188@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:34:54.564225 19930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:54.548174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55486 (local address 127.28.244.190:42537)
0929 20:34:54.549510 (+ 1336us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:54.549516 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:54.549534 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:34:54.549592 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:54.549597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:54.549674 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:54.549785 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:54.550918 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.551678 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.553331 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.553958 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.554106 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:54.557501 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:54.557509 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:54.557514 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:54.557555 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:54.560870 (+ 3315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:54.561768 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:54.561774 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:54.561777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:54.561943 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:54.562389 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:54.562393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:54.562396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:54.562796 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:54.562953 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:54.563526 (+ 573us) server_negotiation.cc:299] Negotiation successful
0929 20:34:54.563786 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1206,"thread_start_us":1073,"threads_started":1}
I20260929 20:34:54.565205 19928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:54.543574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37130 (local address 127.28.244.188:36063)
0929 20:34:54.544306 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:54.544319 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:54.544363 (+ 44us) server_negotiation.cc:407] Connection header received
0929 20:34:54.546318 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:54.546344 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:54.546808 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:54.547190 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:54.549040 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.550713 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.552045 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.552784 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.552993 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:54.557244 (+ 4251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:54.557253 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:54.557266 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:54.557296 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:54.560924 (+ 3628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:54.561784 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:54.561792 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:54.561795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:54.561970 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:54.562394 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:54.562399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:54.562401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:54.563100 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:54.563324 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:54.563751 (+ 427us) server_negotiation.cc:299] Negotiation successful
0929 20:34:54.564184 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":157,"server-negotiator.queue_time_us":535,"thread_start_us":333,"threads_started":1}
I20260929 20:34:54.565248 19929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:54.548144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:42537 (local address 127.0.0.1:55486)
0929 20:34:54.548830 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:54.548876 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:54.549026 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:54.549857 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:54.549861 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:54.550477 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:34:54.550744 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:54.550753 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.551854 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.551858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:54.553156 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:54.553168 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.554191 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.554199 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:54.554354 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:54.554972 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:34:54.555024 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:34:54.557288 (+ 2264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:34:54.561042 (+ 3754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:54.561054 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:54.561067 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:54.561647 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:54.562073 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:54.562078 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:54.562081 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:54.562279 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:54.562926 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:34:54.562933 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:34:54.563130 (+ 197us) client_negotiation.cc:769] Sending connection context
0929 20:34:54.563454 (+ 324us) client_negotiation.cc:240] Negotiation successful
0929 20:34:54.564023 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":42,"thread_start_us":266,"threads_started":1}
I20260929 20:34:54.567071 19925 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.244.189" port: 36903 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:34:54 dist-test-slave-p29l krb5kdc[19696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714094, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.188@KRBTEST.COM for kudu/127.28.244.189@KRBTEST.COM
Sep 29 20:34:54 dist-test-slave-p29l krb5kdc[19696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:34:54.581324 19929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:54.567970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:36903 (local address 127.0.0.1:42966)
0929 20:34:54.568174 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:54.568196 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:54.568303 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:54.568776 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:54.568779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:54.569102 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:34:54.569242 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:54.569249 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.570221 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.570229 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:54.571096 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:54.571107 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.571797 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.571841 (+ 44us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:54.572046 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:54.572650 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:34:54.572668 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:34:54.575030 (+ 2362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:34:54.578631 (+ 3601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:54.578636 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:54.578641 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:54.579175 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:54.579554 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:54.579558 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:54.579563 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:54.579737 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:54.580347 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:34:54.580352 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:34:54.580513 (+ 161us) client_negotiation.cc:769] Sending connection context
0929 20:34:54.580653 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 20:34:54.580958 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 20:34:54.581351 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:54.567986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42966 (local address 127.28.244.189:36903)
0929 20:34:54.568172 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:54.568178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:54.568244 (+ 66us) server_negotiation.cc:407] Connection header received
0929 20:34:54.568467 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:54.568473 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:54.568552 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:54.568609 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:54.569386 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.570086 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.571237 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.571652 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.571789 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:54.575235 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:54.575240 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:54.575243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:54.575278 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:54.578463 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:54.579299 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:54.579303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:54.579305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:54.579437 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:54.579899 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:54.579902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:54.579904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:54.580213 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:54.580376 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:54.580754 (+ 378us) server_negotiation.cc:299] Negotiation successful
0929 20:34:54.580968 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:34:54.582571 19925 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.244.188" port: 36063 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:34:54.582902 19930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:54.568408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55498 (local address 127.28.244.190:42537)
0929 20:34:54.568574 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:54.568579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:54.568597 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:34:54.568782 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:54.568787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:54.568868 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:54.568968 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:54.570117 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.570796 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.572068 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.572494 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.572626 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:54.577129 (+ 4503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:54.577134 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:54.577137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:54.577170 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:54.580170 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:54.580816 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:54.580822 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:54.580826 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:54.580956 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:54.581333 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:54.581338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:54.581341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:54.581733 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:54.582020 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:54.582367 (+ 347us) server_negotiation.cc:299] Negotiation successful
0929 20:34:54.582588 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 20:34:54 dist-test-slave-p29l krb5kdc[19696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714094, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.188@KRBTEST.COM for kudu/127.28.244.188@KRBTEST.COM
Sep 29 20:34:54 dist-test-slave-p29l krb5kdc[19696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.189@KRBTEST.COM
I20260929 20:34:54.595263 19929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:54.583280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:36063 (local address 127.0.0.1:37142)
0929 20:34:54.583465 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:54.583483 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:54.583607 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:54.583932 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:54.583939 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:54.584371 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:34:54.584551 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:54.584564 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.585484 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.585489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:54.586296 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:54.586305 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.586972 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.586977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:54.587182 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:54.587730 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:34:54.587746 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:34:54.589695 (+ 1949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:34:54.592968 (+ 3273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:54.592975 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:54.593001 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:54.593337 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:54.593720 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:54.593727 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:54.593732 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:54.593853 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:54.594347 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:34:54.594352 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:34:54.594501 (+ 149us) client_negotiation.cc:769] Sending connection context
0929 20:34:54.594694 (+ 193us) client_negotiation.cc:240] Negotiation successful
0929 20:34:54.594973 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 20:34:54.595906 19928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:54.583499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37142 (local address 127.28.244.188:36063)
0929 20:34:54.583660 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:54.583667 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:54.583683 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:34:54.583723 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:54.583727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:54.583796 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:54.583900 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:54.584701 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.585368 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.586427 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.586835 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.586965 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:54.589869 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:54.589874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:54.589877 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:54.589903 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:54.592844 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:54.593452 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:54.593460 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:54.593463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:54.593587 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:54.593962 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:54.593966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:54.593969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:54.594235 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:54.594384 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:54.594859 (+ 475us) server_negotiation.cc:299] Negotiation successful
0929 20:34:54.595170 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":57}
I20260929 20:34:54.600296 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:54.584567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42982 (local address 127.28.244.189:36903)
0929 20:34:54.584739 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:54.584745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:54.584782 (+ 37us) server_negotiation.cc:407] Connection header received
0929 20:34:54.584959 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:54.584964 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:54.585042 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:54.585137 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:54.586289 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.587020 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.588436 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.588857 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.588999 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:54.593979 (+ 4980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:54.593986 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:54.593991 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:54.594038 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:54.597487 (+ 3449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:54.598343 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:54.598347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:54.598350 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:54.598534 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:54.598960 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:54.598964 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:54.598966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:54.599234 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:54.599442 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:54.599774 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 20:34:54.600036 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:34:54.601940 19925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00926a6f776c4e50b11cec5b7a4f88d6: Bootstrap starting.
I20260929 20:34:54.605278 29650 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:34:54.605821 19925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00926a6f776c4e50b11cec5b7a4f88d6: Neither blocks nor log segments found. Creating new log.
I20260929 20:34:54.607095 19925 log.cc:824] T 00000000000000000000000000000000 P 00926a6f776c4e50b11cec5b7a4f88d6: Log is configured to *not* fsync() on all Append() calls
I20260929 20:34:54.609961 19925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00926a6f776c4e50b11cec5b7a4f88d6: No bootstrap required, opened a new log
I20260929 20:34:54.608769 19751 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "56c07fd2e1fc4521977b3f9b6e5a4c8f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55498
I20260929 20:34:54.614960 19777 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.188" port: 36063 } attempt: 4
I20260929 20:34:54.617918 19925 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00926a6f776c4e50b11cec5b7a4f88d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c07fd2e1fc4521977b3f9b6e5a4c8f" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 42537 } } peers { permanent_uuid: "642de662dfad45a28e4afc8712f027c2" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 36903 } } peers { permanent_uuid: "00926a6f776c4e50b11cec5b7a4f88d6" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 36063 } }
I20260929 20:34:54.618320 19925 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00926a6f776c4e50b11cec5b7a4f88d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00926a6f776c4e50b11cec5b7a4f88d6, State: Initialized, Role: FOLLOWER
I20260929 20:34:54.618798 19925 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00926a6f776c4e50b11cec5b7a4f88d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c07fd2e1fc4521977b3f9b6e5a4c8f" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 42537 } } peers { permanent_uuid: "642de662dfad45a28e4afc8712f027c2" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 36903 } } peers { permanent_uuid: "00926a6f776c4e50b11cec5b7a4f88d6" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 36063 } }
I20260929 20:34:54.619743 19933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00926a6f776c4e50b11cec5b7a4f88d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c07fd2e1fc4521977b3f9b6e5a4c8f" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 42537 } } peers { permanent_uuid: "642de662dfad45a28e4afc8712f027c2" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 36903 } } peers { permanent_uuid: "00926a6f776c4e50b11cec5b7a4f88d6" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 36063 } } }
I20260929 20:34:54.620225 19933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00926a6f776c4e50b11cec5b7a4f88d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:34:54.621083 19925 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00926a6f776c4e50b11cec5b7a4f88d6 [sys.catalog]: configured and running, proceeding with master startup.
Sep 29 20:34:54 dist-test-slave-p29l krb5kdc[19696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714093, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for kudu/127.28.244.188@KRBTEST.COM
W20260929 20:34:54.631054 19944 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 00926a6f776c4e50b11cec5b7a4f88d6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 20:34:54.631186 19944 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 20:34:54.632895 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:54.615628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:36063 (local address 127.0.0.1:37144)
0929 20:34:54.615854 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:54.615894 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:54.616038 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:54.616415 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:54.616420 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:54.616941 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:34:54.617164 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:54.617177 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.618534 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.618539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:54.620002 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:54.620018 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.621139 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.621146 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:54.621367 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:54.622043 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:34:54.622074 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:34:54.625048 (+ 2974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:34:54.629765 (+ 4717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:54.629774 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:54.629780 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:54.630331 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:54.630821 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:54.630826 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:54.630830 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:54.630997 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:54.631838 (+ 841us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:34:54.631843 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:34:54.632033 (+ 190us) client_negotiation.cc:769] Sending connection context
0929 20:34:54.632178 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 20:34:54.632467 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 20:34:54.633965 19928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:54.615725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37144 (local address 127.28.244.188:36063)
0929 20:34:54.615962 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:54.615969 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:54.616000 (+ 31us) server_negotiation.cc:407] Connection header received
0929 20:34:54.616184 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:54.616190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:54.616266 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:54.616371 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:54.617321 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.618394 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.620177 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.620910 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.621108 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:54.625300 (+ 4192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:54.625309 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:54.625317 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:54.625373 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:54.629582 (+ 4209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:54.630491 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:54.630497 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:54.630501 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:54.630681 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:54.631132 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:54.631137 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:54.631140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:54.631583 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:54.631756 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:54.633306 (+ 1550us) server_negotiation.cc:299] Negotiation successful
0929 20:34:54.633562 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 20:34:54.639848 19777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56c07fd2e1fc4521977b3f9b6e5a4c8f: Bootstrap starting.
I20260929 20:34:54.628782 19858 master_runner.cc:186] Error getting master registration for 127.28.244.190:42537: OK, instance_id { permanent_uuid: "56c07fd2e1fc4521977b3f9b6e5a4c8f" instance_seqno: 1790714093472993 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 20:34:54.640131 19858 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 20:34:54.642946 19777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56c07fd2e1fc4521977b3f9b6e5a4c8f: Neither blocks nor log segments found. Creating new log.
I20260929 20:34:54.644323 19777 log.cc:824] T 00000000000000000000000000000000 P 56c07fd2e1fc4521977b3f9b6e5a4c8f: Log is configured to *not* fsync() on all Append() calls
I20260929 20:34:54.646883 19777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56c07fd2e1fc4521977b3f9b6e5a4c8f: No bootstrap required, opened a new log
I20260929 20:34:54.651886 19777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56c07fd2e1fc4521977b3f9b6e5a4c8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c07fd2e1fc4521977b3f9b6e5a4c8f" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 42537 } } peers { permanent_uuid: "642de662dfad45a28e4afc8712f027c2" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 36903 } } peers { permanent_uuid: "00926a6f776c4e50b11cec5b7a4f88d6" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 36063 } }
I20260929 20:34:54.652184 19777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56c07fd2e1fc4521977b3f9b6e5a4c8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56c07fd2e1fc4521977b3f9b6e5a4c8f, State: Initialized, Role: FOLLOWER
I20260929 20:34:54.652673 19777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56c07fd2e1fc4521977b3f9b6e5a4c8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c07fd2e1fc4521977b3f9b6e5a4c8f" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 42537 } } peers { permanent_uuid: "642de662dfad45a28e4afc8712f027c2" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 36903 } } peers { permanent_uuid: "00926a6f776c4e50b11cec5b7a4f88d6" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 36063 } }
I20260929 20:34:54.654086 19947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56c07fd2e1fc4521977b3f9b6e5a4c8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c07fd2e1fc4521977b3f9b6e5a4c8f" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 42537 } } peers { permanent_uuid: "642de662dfad45a28e4afc8712f027c2" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 36903 } } peers { permanent_uuid: "00926a6f776c4e50b11cec5b7a4f88d6" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 36063 } } }
I20260929 20:34:54.654798 19777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56c07fd2e1fc4521977b3f9b6e5a4c8f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:34:54.654932 19947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56c07fd2e1fc4521977b3f9b6e5a4c8f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:34:54.655014 19751 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56c07fd2e1fc4521977b3f9b6e5a4c8f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 20:34:54.655210 19751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56c07fd2e1fc4521977b3f9b6e5a4c8f [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:34:54.659569 19751 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56c07fd2e1fc4521977b3f9b6e5a4c8f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c07fd2e1fc4521977b3f9b6e5a4c8f" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 42537 } } peers { permanent_uuid: "642de662dfad45a28e4afc8712f027c2" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 36903 } } peers { permanent_uuid: "00926a6f776c4e50b11cec5b7a4f88d6" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 36063 } }
I20260929 20:34:54.663988 19751 leader_election.cc:288] T 00000000000000000000000000000000 P 56c07fd2e1fc4521977b3f9b6e5a4c8f [CANDIDATE]: Term 1 election: Requested vote from peers 642de662dfad45a28e4afc8712f027c2 (127.28.244.189:36903), 00926a6f776c4e50b11cec5b7a4f88d6 (127.28.244.188:36063)
I20260929 20:34:54.663846 19900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "56c07fd2e1fc4521977b3f9b6e5a4c8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "00926a6f776c4e50b11cec5b7a4f88d6"
I20260929 20:34:54.664479 19900 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00926a6f776c4e50b11cec5b7a4f88d6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:34:54.668834 19900 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 00926a6f776c4e50b11cec5b7a4f88d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56c07fd2e1fc4521977b3f9b6e5a4c8f in term 1.
I20260929 20:34:54.669847 19725 leader_election.cc:302] T 00000000000000000000000000000000 P 56c07fd2e1fc4521977b3f9b6e5a4c8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00926a6f776c4e50b11cec5b7a4f88d6, 56c07fd2e1fc4521977b3f9b6e5a4c8f; no voters:
I20260929 20:34:54.670502 19947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56c07fd2e1fc4521977b3f9b6e5a4c8f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:34:54.680464 19947 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56c07fd2e1fc4521977b3f9b6e5a4c8f [term 1 LEADER]: Becoming Leader. State: Replica: 56c07fd2e1fc4521977b3f9b6e5a4c8f, State: Running, Role: LEADER
I20260929 20:34:54.681425 19947 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56c07fd2e1fc4521977b3f9b6e5a4c8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c07fd2e1fc4521977b3f9b6e5a4c8f" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 42537 } } peers { permanent_uuid: "642de662dfad45a28e4afc8712f027c2" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 36903 } } peers { permanent_uuid: "00926a6f776c4e50b11cec5b7a4f88d6" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 36063 } }
I20260929 20:34:54.687281 19958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56c07fd2e1fc4521977b3f9b6e5a4c8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56c07fd2e1fc4521977b3f9b6e5a4c8f. Latest consensus state: current_term: 1 leader_uuid: "56c07fd2e1fc4521977b3f9b6e5a4c8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c07fd2e1fc4521977b3f9b6e5a4c8f" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 42537 } } peers { permanent_uuid: "642de662dfad45a28e4afc8712f027c2" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 36903 } } peers { permanent_uuid: "00926a6f776c4e50b11cec5b7a4f88d6" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 36063 } } }
I20260929 20:34:54.688020 19958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56c07fd2e1fc4521977b3f9b6e5a4c8f [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:54.692602 19960 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260929 20:34:54.711093 19730 tablet.cc:2369] T 00000000000000000000000000000000 P 56c07fd2e1fc4521977b3f9b6e5a4c8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:34:54.714246 19960 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:34:54.719280 19851 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.188" port: 36063 } attempt: 5
Sep 29 20:34:54 dist-test-slave-p29l krb5kdc[19696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714093, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.189@KRBTEST.COM for kudu/127.28.244.188@KRBTEST.COM
I20260929 20:34:54.740413 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:54.719912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:36063 (local address 127.0.0.1:37152)
0929 20:34:54.720363 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:54.720386 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:54.720515 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:54.721602 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:54.721607 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:54.722152 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:34:54.722390 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:54.722407 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.724447 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.724456 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:54.726546 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:54.726567 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.728149 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.728155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:54.728434 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:54.729247 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:34:54.729278 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:34:54.732704 (+ 3426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:34:54.737246 (+ 4542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:54.737251 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:54.737255 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:54.737797 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:54.738295 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:54.738302 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:54.738307 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:54.738482 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:54.739228 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:34:54.739234 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:34:54.739439 (+ 205us) client_negotiation.cc:769] Sending connection context
0929 20:34:54.739615 (+ 176us) client_negotiation.cc:240] Negotiation successful
0929 20:34:54.739979 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295}
I20260929 20:34:54.741518 19928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:54.720818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37152 (local address 127.28.244.188:36063)
0929 20:34:54.721071 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:54.721078 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:54.721097 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:34:54.721163 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:54.721170 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:54.721295 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:54.721401 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:54.722662 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.724204 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.726763 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.727742 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.727955 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:54.732977 (+ 5022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:54.732985 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:54.732989 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:54.733032 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:54.737041 (+ 4009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:54.737967 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:54.737973 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:54.737977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:54.738149 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:54.738622 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:54.738630 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:54.738634 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:54.739008 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:54.739152 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:54.740808 (+ 1656us) server_negotiation.cc:299] Negotiation successful
0929 20:34:54.741096 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260929 20:34:54.747474 19851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 642de662dfad45a28e4afc8712f027c2: Bootstrap starting.
I20260929 20:34:54.751780 19851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 642de662dfad45a28e4afc8712f027c2: Neither blocks nor log segments found. Creating new log.
I20260929 20:34:54.756130 19930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:54.707175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55500 (local address 127.28.244.190:42537)
0929 20:34:54.707376 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:54.707384 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:54.714567 (+ 7183us) server_negotiation.cc:407] Connection header received
0929 20:34:54.715187 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:54.715197 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:54.715307 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:54.715397 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:54.718023 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.719187 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.743564 (+ 24377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.744276 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.744382 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:54.747843 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:54.747851 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:54.747855 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:54.747908 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:54.751728 (+ 3820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:54.752710 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:54.752718 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:54.752721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:54.752924 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:54.753491 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:54.753496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:54.753499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:54.754582 (+ 1083us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:54.754856 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:54.755417 (+ 561us) server_negotiation.cc:299] Negotiation successful
0929 20:34:54.755662 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 20:34:54.761541 19851 log.cc:824] T 00000000000000000000000000000000 P 642de662dfad45a28e4afc8712f027c2: Log is configured to *not* fsync() on all Append() calls
W20260929 20:34:54.764808 19879 tablet.cc:2369] T 00000000000000000000000000000000 P 00926a6f776c4e50b11cec5b7a4f88d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:34:54.771390 19900 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 00926a6f776c4e50b11cec5b7a4f88d6 [term 1 FOLLOWER]: Refusing update from remote peer 56c07fd2e1fc4521977b3f9b6e5a4c8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:34:54.772456 19947 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 56c07fd2e1fc4521977b3f9b6e5a4c8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "00926a6f776c4e50b11cec5b7a4f88d6" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 36063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:34:54.757956 19709 master_runner.cc:186] Error getting master registration for 127.28.244.189:36903: OK, instance_id { permanent_uuid: "642de662dfad45a28e4afc8712f027c2" instance_seqno: 1790714093988088 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 20:34:54.788975 19709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 20:34:54.801219 19969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:54.722032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37158 (local address 127.28.244.188:36063)
0929 20:34:54.734101 (+ 12069us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:54.734110 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:54.734138 (+ 28us) server_negotiation.cc:407] Connection header received
0929 20:34:54.734192 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:54.734203 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:54.734275 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:54.734411 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:54.740871 (+ 6460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.761073 (+ 20202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.782123 (+ 21050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.782880 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.783024 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:54.789526 (+ 6502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:54.789534 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:54.789540 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:54.789587 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:54.793953 (+ 4366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:54.795960 (+ 2007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:54.795968 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:54.796043 (+ 75us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:54.796412 (+ 369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:54.797394 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:54.797399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:54.797403 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:54.798151 (+ 748us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:54.798669 (+ 518us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:54.799792 (+ 1123us) server_negotiation.cc:299] Negotiation successful
0929 20:34:54.800321 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11899,"thread_start_us":12605,"threads_started":1}
I20260929 20:34:54.805457 19933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00926a6f776c4e50b11cec5b7a4f88d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56c07fd2e1fc4521977b3f9b6e5a4c8f. Latest consensus state: current_term: 1 leader_uuid: "56c07fd2e1fc4521977b3f9b6e5a4c8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c07fd2e1fc4521977b3f9b6e5a4c8f" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 42537 } } peers { permanent_uuid: "642de662dfad45a28e4afc8712f027c2" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 36903 } } peers { permanent_uuid: "00926a6f776c4e50b11cec5b7a4f88d6" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 36063 } } }
I20260929 20:34:54.805923 19933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00926a6f776c4e50b11cec5b7a4f88d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:34:54.808810 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:54.706859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42998 (local address 127.28.244.189:36903)
0929 20:34:54.707085 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:54.707091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:54.721553 (+ 14462us) server_negotiation.cc:407] Connection header received
0929 20:34:54.737002 (+ 15449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:54.737007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:54.737107 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:54.737176 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:54.753855 (+ 16679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.754912 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.779783 (+ 24871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:54.780547 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:54.780689 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:54.795192 (+ 14503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:54.795200 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:54.795205 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:54.795252 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:54.799494 (+ 4242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:54.801057 (+ 1563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:54.801063 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:54.801066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:54.801225 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:54.802020 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:54.802025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:54.802028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:54.802397 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:54.802625 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:54.808154 (+ 5529us) server_negotiation.cc:299] Negotiation successful
0929 20:34:54.808432 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 20:34:54.809669 19972 mvcc.cc:205] Tried to move back new op lower bound from 7334764932168654848 to 7334764931823722496. Current Snapshot: MvccSnapshot[applied={T|T < 7334764932168654848}]
I20260929 20:34:54.813221 19974 mvcc.cc:205] Tried to move back new op lower bound from 7334764932168654848 to 7334764931823722496. Current Snapshot: MvccSnapshot[applied={T|T < 7334764932168654848}]
I20260929 20:34:54.815135 19851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 642de662dfad45a28e4afc8712f027c2: No bootstrap required, opened a new log
I20260929 20:34:54.817804 19958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56c07fd2e1fc4521977b3f9b6e5a4c8f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "56c07fd2e1fc4521977b3f9b6e5a4c8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c07fd2e1fc4521977b3f9b6e5a4c8f" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 42537 } } peers { permanent_uuid: "642de662dfad45a28e4afc8712f027c2" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 36903 } } peers { permanent_uuid: "00926a6f776c4e50b11cec5b7a4f88d6" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 36063 } } }
I20260929 20:34:54.818104 19958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56c07fd2e1fc4521977b3f9b6e5a4c8f [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:54.820369 19933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00926a6f776c4e50b11cec5b7a4f88d6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "56c07fd2e1fc4521977b3f9b6e5a4c8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c07fd2e1fc4521977b3f9b6e5a4c8f" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 42537 } } peers { permanent_uuid: "642de662dfad45a28e4afc8712f027c2" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 36903 } } peers { permanent_uuid: "00926a6f776c4e50b11cec5b7a4f88d6" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 36063 } } }
I20260929 20:34:54.820608 19933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00926a6f776c4e50b11cec5b7a4f88d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:34:54.822074 19960 catalog_manager.cc:1347] Generated new cluster ID: c0b5bdbda3584dfcae552bf4479b605c
I20260929 20:34:54.822170 19960 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:34:54.827215 19851 raft_consensus.cc:348] T 00000000000000000000000000000000 P 642de662dfad45a28e4afc8712f027c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c07fd2e1fc4521977b3f9b6e5a4c8f" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 42537 } } peers { permanent_uuid: "642de662dfad45a28e4afc8712f027c2" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 36903 } } peers { permanent_uuid: "00926a6f776c4e50b11cec5b7a4f88d6" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 36063 } }
I20260929 20:34:54.831369 19851 raft_consensus.cc:729] T 00000000000000000000000000000000 P 642de662dfad45a28e4afc8712f027c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 642de662dfad45a28e4afc8712f027c2, State: Initialized, Role: FOLLOWER
I20260929 20:34:54.832015 19851 consensus_queue.cc:260] T 00000000000000000000000000000000 P 642de662dfad45a28e4afc8712f027c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c07fd2e1fc4521977b3f9b6e5a4c8f" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 42537 } } peers { permanent_uuid: "642de662dfad45a28e4afc8712f027c2" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 36903 } } peers { permanent_uuid: "00926a6f776c4e50b11cec5b7a4f88d6" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 36063 } }
I20260929 20:34:54.839090 19978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 642de662dfad45a28e4afc8712f027c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c07fd2e1fc4521977b3f9b6e5a4c8f" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 42537 } } peers { permanent_uuid: "642de662dfad45a28e4afc8712f027c2" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 36903 } } peers { permanent_uuid: "00926a6f776c4e50b11cec5b7a4f88d6" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 36063 } } }
I20260929 20:34:54.841460 19978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 642de662dfad45a28e4afc8712f027c2 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:34:54.841336 19851 sys_catalog.cc:564] T 00000000000000000000000000000000 P 642de662dfad45a28e4afc8712f027c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:34:54.841606 19826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "56c07fd2e1fc4521977b3f9b6e5a4c8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "642de662dfad45a28e4afc8712f027c2"
I20260929 20:34:54.842105 19826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 642de662dfad45a28e4afc8712f027c2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:34:54.847263 19826 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 642de662dfad45a28e4afc8712f027c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56c07fd2e1fc4521977b3f9b6e5a4c8f in term 1.
I20260929 20:34:54.848260 19825 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 642de662dfad45a28e4afc8712f027c2 [term 1 FOLLOWER]: Refusing update from remote peer 56c07fd2e1fc4521977b3f9b6e5a4c8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:34:54.850224 19947 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 56c07fd2e1fc4521977b3f9b6e5a4c8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "642de662dfad45a28e4afc8712f027c2" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 36903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:34:54.886176 19980 mvcc.cc:205] Tried to move back new op lower bound from 7334764932168654848 to 7334764931823722496. Current Snapshot: MvccSnapshot[applied={T|T < 7334764932168654848}]
I20260929 20:34:54.887466 19978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 642de662dfad45a28e4afc8712f027c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56c07fd2e1fc4521977b3f9b6e5a4c8f. Latest consensus state: current_term: 1 leader_uuid: "56c07fd2e1fc4521977b3f9b6e5a4c8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c07fd2e1fc4521977b3f9b6e5a4c8f" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 42537 } } peers { permanent_uuid: "642de662dfad45a28e4afc8712f027c2" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 36903 } } peers { permanent_uuid: "00926a6f776c4e50b11cec5b7a4f88d6" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 36063 } } }
I20260929 20:34:54.887777 19978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 642de662dfad45a28e4afc8712f027c2 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:34:54.888185 19982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 642de662dfad45a28e4afc8712f027c2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "56c07fd2e1fc4521977b3f9b6e5a4c8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c07fd2e1fc4521977b3f9b6e5a4c8f" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 42537 } } peers { permanent_uuid: "642de662dfad45a28e4afc8712f027c2" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 36903 } } peers { permanent_uuid: "00926a6f776c4e50b11cec5b7a4f88d6" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 36063 } } }
I20260929 20:34:54.889115 19982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 642de662dfad45a28e4afc8712f027c2 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:34:54.902022 19947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56c07fd2e1fc4521977b3f9b6e5a4c8f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "56c07fd2e1fc4521977b3f9b6e5a4c8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c07fd2e1fc4521977b3f9b6e5a4c8f" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 42537 } } peers { permanent_uuid: "642de662dfad45a28e4afc8712f027c2" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 36903 } } peers { permanent_uuid: "00926a6f776c4e50b11cec5b7a4f88d6" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 36063 } } }
I20260929 20:34:54.902407 19947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56c07fd2e1fc4521977b3f9b6e5a4c8f [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:54.912696 19960 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:34:54.914657 19960 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:34:54.947764 19960 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56c07fd2e1fc4521977b3f9b6e5a4c8f: Generated new TSK 0
I20260929 20:34:54.948961 19960 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:34:54.970984 19993 catalog_manager.cc:1259] Loaded cluster ID: c0b5bdbda3584dfcae552bf4479b605c
I20260929 20:34:54.971103 19993 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 642de662dfad45a28e4afc8712f027c2: loading cluster ID for follower catalog manager: success
I20260929 20:34:54.974896 19993 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 642de662dfad45a28e4afc8712f027c2: acquiring CA information for follower catalog manager: success
I20260929 20:34:54.978446 19993 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 642de662dfad45a28e4afc8712f027c2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 20:34:55.081154 29650 mini_kdc.cc:348] Time spent kdestroy: real 0.106s user 0.000s sys 0.022s
W20260929 20:34:55.150738 19930 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:34:55.134203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55514 (local address 127.28.244.190:42537)
0929 20:34:55.134470 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:55.134479 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:55.148568 (+ 14089us) server_negotiation.cc:407] Connection header received
0929 20:34:55.148926 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:55.148937 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:55.149051 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:55.149157 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:55.150450 (+ 1293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55514: BlockingRecv error: recv got EOF from 127.0.0.1:55514 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260929 20:34:55.153329 19855 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:34:55.146311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43010 (local address 127.28.244.189:36903)
0929 20:34:55.146593 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:55.146603 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:55.151350 (+ 4747us) server_negotiation.cc:407] Connection header received
0929 20:34:55.151606 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:55.151612 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:55.151727 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:55.151803 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:55.153005 (+ 1202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43010: BlockingRecv error: recv got EOF from 127.0.0.1:43010 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260929 20:34:55.155519 19969 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:34:55.148614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37164 (local address 127.28.244.188:36063)
0929 20:34:55.148802 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:55.148810 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:55.153949 (+ 5139us) server_negotiation.cc:407] Connection header received
0929 20:34:55.154113 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:55.154117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:55.154224 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:55.154331 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:55.155268 (+ 937us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37164: BlockingRecv error: recv got EOF from 127.0.0.1:37164 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 20:34:55.216184 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:55.195464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43014 (local address 127.28.244.189:36903)
0929 20:34:55.195670 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:55.195676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:55.208469 (+ 12793us) server_negotiation.cc:407] Connection header received
0929 20:34:55.208762 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:55.208768 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:55.208976 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:55.209113 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:34:55.209863 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:55.210992 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:55.213061 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:55.213729 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:55.213832 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:55.214638 (+ 806us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:34:55.215342 (+ 704us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:34:55.215459 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:55.215707 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 20:34:55.215830 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260929 20:34:55.262660 19969 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:34:55.249789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37180 (local address 127.28.244.188:36063)
0929 20:34:55.249989 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:55.249995 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:55.260680 (+ 10685us) server_negotiation.cc:407] Connection header received
0929 20:34:55.260969 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:55.260975 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:55.261129 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:55.261218 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:55.262387 (+ 1169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37180: BlockingRecv error: recv got EOF from 127.0.0.1:37180 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 20:34:55.284008 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:55.276113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43026 (local address 127.28.244.189:36903)
0929 20:34:55.276312 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:55.276320 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:55.276884 (+ 564us) server_negotiation.cc:407] Connection header received
0929 20:34:55.277258 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:55.277265 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:55.277367 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:55.277445 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:34:55.278202 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:55.279395 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:55.281527 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:55.282253 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:55.282358 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:55.283052 (+ 694us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:34:55.283235 (+ 183us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:34:55.283303 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:55.283552 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 20:34:55.283642 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260929 20:34:55.294905 19969 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:34:55.292246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37194 (local address 127.28.244.188:36063)
0929 20:34:55.292427 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:55.292433 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:55.293104 (+ 671us) server_negotiation.cc:407] Connection header received
0929 20:34:55.293402 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:55.293408 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:55.293503 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:55.293580 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:55.294644 (+ 1064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37194: BlockingRecv error: recv got EOF from 127.0.0.1:37194 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:34:55.308413 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:55.301144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43042 (local address 127.28.244.189:36903)
0929 20:34:55.301341 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:55.301347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:55.301827 (+ 480us) server_negotiation.cc:407] Connection header received
0929 20:34:55.302159 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:55.302165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:55.302265 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:55.302342 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:34:55.303048 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:55.304288 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:55.306115 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:55.306836 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:55.306998 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:55.307504 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:34:55.307668 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:34:55.307766 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:55.307973 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 20:34:55.308068 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260929 20:34:55.318228 19969 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:34:55.315639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37206 (local address 127.28.244.188:36063)
0929 20:34:55.315891 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:55.315898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:55.316422 (+ 524us) server_negotiation.cc:407] Connection header received
0929 20:34:55.316729 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:55.316735 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:55.316833 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:55.316932 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:55.317972 (+ 1040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37206: BlockingRecv error: recv got EOF from 127.0.0.1:37206 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 20:34:55.334457 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:55.326819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43048 (local address 127.28.244.189:36903)
0929 20:34:55.327033 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:55.327040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:55.327594 (+ 554us) server_negotiation.cc:407] Connection header received
0929 20:34:55.327953 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:55.327962 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:55.328060 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:55.328128 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:34:55.328809 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:55.330070 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:55.332134 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:55.332896 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:55.333006 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:55.333631 (+ 625us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:34:55.333775 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:34:55.333838 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:55.334075 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 20:34:55.334162 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260929 20:34:55.346215 19969 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:34:55.342455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37220 (local address 127.28.244.188:36063)
0929 20:34:55.343202 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:55.343209 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:55.344354 (+ 1145us) server_negotiation.cc:407] Connection header received
0929 20:34:55.344670 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:55.344676 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:55.344791 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:55.344919 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:55.345967 (+ 1048us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37220: BlockingRecv error: recv got EOF from 127.0.0.1:37220 (error 108)
Metrics: {"server-negotiator.queue_time_us":600}
I20260929 20:34:55.367370 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:55.358957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43056 (local address 127.28.244.189:36903)
0929 20:34:55.359202 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:55.359209 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:55.359970 (+ 761us) server_negotiation.cc:407] Connection header received
0929 20:34:55.360302 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:55.360308 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:55.360413 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:55.360489 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:34:55.361225 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:55.362440 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:55.364541 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:55.365341 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:55.365474 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:55.366348 (+ 874us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:34:55.366564 (+ 216us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:34:55.366636 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:55.366900 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 20:34:55.366991 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260929 20:34:55.378639 19969 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:34:55.376123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37228 (local address 127.28.244.188:36063)
0929 20:34:55.376319 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:55.376326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:55.376799 (+ 473us) server_negotiation.cc:407] Connection header received
0929 20:34:55.377130 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:55.377136 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:55.377238 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:55.377330 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:55.378372 (+ 1042us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37228: BlockingRecv error: recv got EOF from 127.0.0.1:37228 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:34:55.406916 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:55.399174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43058 (local address 127.28.244.189:36903)
0929 20:34:55.399380 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:55.399389 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:55.399936 (+ 547us) server_negotiation.cc:407] Connection header received
0929 20:34:55.400646 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:55.400655 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:55.400743 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:55.400809 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:34:55.401437 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:55.402622 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:55.404650 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:55.405337 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:55.405438 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:55.406077 (+ 639us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:34:55.406213 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:34:55.406271 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:55.406494 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 20:34:55.406596 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 20:34:55.416725 19969 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:34:55.414199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37230 (local address 127.28.244.188:36063)
0929 20:34:55.414396 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:55.414405 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:55.414951 (+ 546us) server_negotiation.cc:407] Connection header received
0929 20:34:55.415248 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:55.415254 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:55.415348 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:55.415421 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:55.416506 (+ 1085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37230: BlockingRecv error: recv got EOF from 127.0.0.1:37230 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:34:55.463728 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:55.454121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43060 (local address 127.28.244.189:36903)
0929 20:34:55.455042 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:55.455050 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:55.456374 (+ 1324us) server_negotiation.cc:407] Connection header received
0929 20:34:55.456783 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:55.456791 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:55.456927 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:55.456996 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:34:55.457851 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:55.459330 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:55.461546 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:55.462255 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:55.462413 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:55.462842 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:34:55.463000 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:34:55.463099 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:55.463316 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 20:34:55.463411 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714}
W20260929 20:34:55.472177 19969 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:34:55.470130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37236 (local address 127.28.244.188:36063)
0929 20:34:55.470339 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:55.470346 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:55.470729 (+ 383us) server_negotiation.cc:407] Connection header received
0929 20:34:55.470892 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:55.470898 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:55.470999 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:55.471115 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:55.471924 (+ 809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37236: BlockingRecv error: recv got EOF from 127.0.0.1:37236 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 20:34:55.547093 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:55.540692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43062 (local address 127.28.244.189:36903)
0929 20:34:55.540957 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:55.540964 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:55.541334 (+ 370us) server_negotiation.cc:407] Connection header received
0929 20:34:55.541533 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:55.541537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:55.541610 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:55.541792 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:34:55.542212 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:55.543007 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:55.545077 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:55.545607 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:55.545748 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:55.546214 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:34:55.546353 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:34:55.546432 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:55.546655 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 20:34:55.546730 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260929 20:34:55.556660 19969 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:34:55.554399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37244 (local address 127.28.244.188:36063)
0929 20:34:55.554688 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:55.554698 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:55.555089 (+ 391us) server_negotiation.cc:407] Connection header received
0929 20:34:55.555252 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:55.555257 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:55.555359 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:55.555458 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:55.556425 (+ 967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37244: BlockingRecv error: recv got EOF from 127.0.0.1:37244 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 20:34:55.633255 19944 catalog_manager.cc:1259] Loaded cluster ID: c0b5bdbda3584dfcae552bf4479b605c
I20260929 20:34:55.633361 19944 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 00926a6f776c4e50b11cec5b7a4f88d6: loading cluster ID for follower catalog manager: success
I20260929 20:34:55.635965 19944 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 00926a6f776c4e50b11cec5b7a4f88d6: acquiring CA information for follower catalog manager: success
I20260929 20:34:55.637953 19944 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 00926a6f776c4e50b11cec5b7a4f88d6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 20:34:55.693995 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:55.688750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43066 (local address 127.28.244.189:36903)
0929 20:34:55.689056 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:55.689064 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:55.689342 (+ 278us) server_negotiation.cc:407] Connection header received
0929 20:34:55.689603 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:55.689607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:55.689677 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:55.689781 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:34:55.690202 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:55.690901 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:55.692222 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:55.692695 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:55.692845 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:55.693256 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:34:55.693376 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:34:55.693475 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:55.693671 (+ 196us) server_negotiation.cc:299] Negotiation successful
0929 20:34:55.693759 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260929 20:34:55.705911 19969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:55.700274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37248 (local address 127.28.244.188:36063)
0929 20:34:55.700442 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:55.700447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:55.700821 (+ 374us) server_negotiation.cc:407] Connection header received
0929 20:34:55.700972 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:55.700976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:55.701134 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:55.701234 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:34:55.701689 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:55.702435 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:55.703802 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:55.704235 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:55.704326 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:55.704834 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:34:55.705378 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:34:55.705483 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:55.705612 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 20:34:55.705680 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 20:34:55.720191 19969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:55.713102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37250 (local address 127.28.244.188:36063)
0929 20:34:55.713292 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:55.713299 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:55.714093 (+ 794us) server_negotiation.cc:407] Connection header received
0929 20:34:55.714245 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:55.714251 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:55.714342 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:55.714427 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:34:55.715187 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:55.716179 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:55.717913 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:55.718564 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:55.718690 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:55.719293 (+ 603us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:34:55.719456 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:34:55.719548 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:55.719786 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 20:34:55.719907 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:34:55.720199 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:55.712262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43082 (local address 127.28.244.189:36903)
0929 20:34:55.712548 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:55.712555 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:55.713512 (+ 957us) server_negotiation.cc:407] Connection header received
0929 20:34:55.713790 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:55.713798 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:55.713891 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:55.713957 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:34:55.714790 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:55.715837 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:55.718087 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:55.718826 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:55.718952 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:55.719440 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:34:55.719581 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:34:55.719665 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:55.719873 (+ 208us) server_negotiation.cc:299] Negotiation successful
0929 20:34:55.719951 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 20:34:55.720628 20100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:55.712308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55516 (local address 127.28.244.190:42537)
0929 20:34:55.712922 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:55.712928 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:55.712950 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:34:55.713313 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:55.713319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:55.713586 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:55.713781 (+ 195us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:34:55.714509 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:55.715510 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:55.718070 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:55.718489 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:55.718636 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:55.719175 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:34:55.719676 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:34:55.719772 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:55.720196 (+ 424us) server_negotiation.cc:299] Negotiation successful
0929 20:34:55.720314 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":349,"threads_started":1}
I20260929 20:34:55.726260 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 19709
I20260929 20:34:55.727301 19772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:34:55.841233 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 19709
W20260929 20:34:55.863454 19876 connection.cc:535] server connection from 127.0.0.1:37144 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 20:34:55.864166 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 19784
I20260929 20:34:55.865365 19846 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:34:55.975605 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 19784
I20260929 20:34:55.996733 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 19858
I20260929 20:34:55.997900 19920 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:34:56.106271 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 19858
2026-09-29T20:34:56Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4480 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:34:56 dist-test-slave-p29l krb5kdc[20109](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:34:56 dist-test-slave-p29l krb5kdc[20109](info): set up 2 sockets
Sep 29 20:34:56 dist-test-slave-p29l krb5kdc[20109](info): commencing operation
krb5kdc: starting...
W20260929 20:34:57.236127 29650 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.006s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/krb5kdc/test-user.keytab.
Sep 29 20:34:57 dist-test-slave-p29l krb5kdc[20109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:34:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:34:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.244.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
Entry for principal kudu/127.28.244.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab.
I20260929 20:34:57.515067 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:46381
--webserver_interface=127.28.244.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:39919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:46381,127.28.244.189:38609,127.28.244.188:37651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:34:57.859758 20122 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 20:34:57.860212 20122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:34:57.860337 20122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:34:57.860472 20122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:34:57.868016 20122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:34:57.868152 20122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:34:57.868263 20122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:34:57.868355 20122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:34:57.868470 20122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:34:57.878878 20122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:39919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:46381,127.28.244.189:38609,127.28.244.188:37651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.190.keytab
--principal=kudu/127.28.244.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.190:46381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.244.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:34:57.880481 20122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:34:57.881861 20122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:34:57.894157 20130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:57.894093 20128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:57.894189 20127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:34:57.894987 20122 server_base.cc:1034] running on GCE node
I20260929 20:34:57.895792 20122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:34:57.897316 20122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:34:57.898602 20122 hybrid_clock.cc:648] HybridClock initialized: now 1790714097898551 us; error 52 us; skew 500 ppm
I20260929 20:34:57.898957 20122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:34:57 dist-test-slave-p29l krb5kdc[20109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714097, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:34:57.902307 20122 init.cc:377] Logged in from keytab as kudu/127.28.244.190@KRBTEST.COM (short username kudu)
I20260929 20:34:57.903708 20122 webserver.cc:460] Webserver started at http://127.28.244.190:34643/ using document root <none> and password file <none>
I20260929 20:34:57.904440 20122 fs_manager.cc:359] Metadata directory not provided
I20260929 20:34:57.904529 20122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:34:57.904800 20122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:34:57.907796 20122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-0/data/instance:
uuid: "c6c8af3248b54a3c85fc56aa28854284"
format_stamp: "Formatted at 2026-09-29 20:34:57 on dist-test-slave-p29l"
server_key: "8a822ba241f4d3295dba4cefa161909e"
server_key_iv: "ded90bf23603943bc5768b467266993f"
server_key_version: "encryptionkey@0"
I20260929 20:34:57.908671 20122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-0/wal/instance:
uuid: "c6c8af3248b54a3c85fc56aa28854284"
format_stamp: "Formatted at 2026-09-29 20:34:57 on dist-test-slave-p29l"
server_key: "8a822ba241f4d3295dba4cefa161909e"
server_key_iv: "ded90bf23603943bc5768b467266993f"
server_key_version: "encryptionkey@0"
I20260929 20:34:57.914028 20122 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 20:34:57.917667 20137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:57.919170 20122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:34:57.919332 20122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-0/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-0/wal
uuid: "c6c8af3248b54a3c85fc56aa28854284"
format_stamp: "Formatted at 2026-09-29 20:34:57 on dist-test-slave-p29l"
server_key: "8a822ba241f4d3295dba4cefa161909e"
server_key_iv: "ded90bf23603943bc5768b467266993f"
server_key_version: "encryptionkey@0"
I20260929 20:34:57.919543 20122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:34:57.938477 20122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:34:57.942893 20122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:34:57.943148 20122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:34:57.962217 20122 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.190:46381
I20260929 20:34:57.962237 20188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.190:46381 every 8 connection(s)
I20260929 20:34:57.963651 20122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-0/data/info.pb
I20260929 20:34:57.967670 20189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:34:57.972147 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 20122
I20260929 20:34:57.972342 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-0/wal/instance
I20260929 20:34:57.972899 29650 external_mini_cluster.cc:1428] Setting key a0a801886bdef903779066c58b4bbab4
I20260929 20:34:57.975697 20189 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.244.190" port: 46381 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:34:57 dist-test-slave-p29l krb5kdc[20109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714097, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.244.189@KRBTEST.COM; defaulting to no policy
I20260929 20:34:58.001047 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:57.979928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:46381 (local address 127.0.0.1:40070)
0929 20:34:57.981208 (+ 1280us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:57.981254 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:57.983084 (+ 1830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:57.984108 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:57.984113 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:57.984706 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:34:57.985074 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:57.985093 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:57.986699 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:57.986704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:57.987868 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:57.987886 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:57.988841 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:57.988848 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:57.989027 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:57.989737 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:34:57.989761 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:34:57.992427 (+ 2666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:34:57.996140 (+ 3713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:57.996155 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:57.996169 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:57.996828 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:57.997284 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:57.997291 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:57.997296 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:57.997537 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:57.998426 (+ 889us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:34:57.998438 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:34:57.998712 (+ 274us) client_negotiation.cc:769] Sending connection context
0929 20:34:57.999041 (+ 329us) client_negotiation.cc:240] Negotiation successful
0929 20:34:57.999868 (+ 827us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1047,"mutex_wait_us":160,"spinlock_wait_cycles":72064,"thread_start_us":807,"threads_started":1}
I20260929 20:34:58.001086 20191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:57.979894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40070 (local address 127.28.244.190:46381)
0929 20:34:57.980806 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:57.980822 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:57.981399 (+ 577us) server_negotiation.cc:407] Connection header received
0929 20:34:57.983485 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:57.983495 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:57.983883 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:57.984058 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:57.985303 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:57.986534 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:57.988012 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:57.988585 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:57.988803 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:57.992702 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:57.992710 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:57.992724 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:57.992755 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:57.995989 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:57.996999 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:57.997006 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:57.997008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:57.997159 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:57.997674 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:57.997678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:57.997680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:57.998265 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:57.998433 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:57.999213 (+ 780us) server_negotiation.cc:299] Negotiation successful
0929 20:34:57.999635 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":384,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.189@KRBTEST.COM" created.
I20260929 20:34:58.004323 20189 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.244.189" port: 38609 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:34:58.005542 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:58.005005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:38609 (local address 127.0.0.1:40828)
0929 20:34:58.005219 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:58.005366 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.189:38609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260929 20:34:58.005842 20138 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.244.189:38609: connect: Connection refused (error 111)
I20260929 20:34:58.007622 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:58.007161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:38609 (local address 127.0.0.1:40840)
0929 20:34:58.007341 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:58.007440 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.189:38609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 20:34:58.007905 20189 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.189:38609: Network error: Client connection negotiation failed: client connection to 127.28.244.189:38609: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.189.keytab.
Entry for principal kudu/127.28.244.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.189.keytab.
I20260929 20:34:58.035682 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.189:38609
--webserver_interface=127.28.244.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.189.keytab
--principal=kudu/127.28.244.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:39919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:46381,127.28.244.189:38609,127.28.244.188:37651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:34:58.057289 20189 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.189" port: 38609 } attempt: 1
I20260929 20:34:58.058513 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:58.058036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:38609 (local address 127.0.0.1:40848)
0929 20:34:58.058243 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:58.058373 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.189:38609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 20:34:58.059502 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:58.059138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:38609 (local address 127.0.0.1:40864)
0929 20:34:58.059285 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:58.059358 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.189:38609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 20:34:58.059679 20189 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.189:38609: Network error: Client connection negotiation failed: client connection to 127.28.244.189:38609: connect: Connection refused (error 111)
I20260929 20:34:58.127944 20189 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.189" port: 38609 } attempt: 2
I20260929 20:34:58.129262 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:58.128710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:38609 (local address 127.0.0.1:40868)
0929 20:34:58.129027 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:58.129120 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.189:38609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260929 20:34:58.130339 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:58.129965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:38609 (local address 127.0.0.1:40872)
0929 20:34:58.130137 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:58.130210 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.189:38609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 20:34:58.130524 20189 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.189:38609: Network error: Client connection negotiation failed: client connection to 127.28.244.189:38609: connect: Connection refused (error 111)
I20260929 20:34:58.221813 20189 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.189" port: 38609 } attempt: 3
I20260929 20:34:58.223114 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:58.222616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:38609 (local address 127.0.0.1:40878)
0929 20:34:58.222866 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:58.222970 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.189:38609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 20:34:58.224189 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:58.223775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:38609 (local address 127.0.0.1:40890)
0929 20:34:58.223962 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:58.224050 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.189:38609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260929 20:34:58.224377 20189 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.189:38609: Network error: Client connection negotiation failed: client connection to 127.28.244.189:38609: connect: Connection refused (error 111)
I20260929 20:34:58.367646 20189 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.189" port: 38609 } attempt: 4
I20260929 20:34:58.368933 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:58.368382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:38609 (local address 127.0.0.1:40906)
0929 20:34:58.368716 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:58.368801 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.189:38609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
I20260929 20:34:58.369927 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:58.369579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:38609 (local address 127.0.0.1:40912)
0929 20:34:58.369725 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:58.369811 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.189:38609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 20:34:58.370155 20189 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.189:38609: Network error: Client connection negotiation failed: client connection to 127.28.244.189:38609: connect: Connection refused (error 111)
W20260929 20:34:58.382689 20196 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 20:34:58.383108 20196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:34:58.383240 20196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:34:58.383386 20196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:34:58.390969 20196 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:34:58.391095 20196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:34:58.391199 20196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:34:58.391295 20196 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:34:58.391383 20196 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:34:58.401654 20196 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:39919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:46381,127.28.244.189:38609,127.28.244.188:37651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.189.keytab
--principal=kudu/127.28.244.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.189:38609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.244.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:34:58.403050 20196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:34:58.404433 20196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:34:58.416316 20201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:58.416383 20204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:58.416268 20202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:34:58.417825 20196 server_base.cc:1034] running on GCE node
I20260929 20:34:58.418670 20196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:34:58.420219 20196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:34:58.421495 20196 hybrid_clock.cc:648] HybridClock initialized: now 1790714098421431 us; error 38 us; skew 500 ppm
I20260929 20:34:58.421876 20196 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:34:58 dist-test-slave-p29l krb5kdc[20109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714098, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:34:58.425284 20196 init.cc:377] Logged in from keytab as kudu/127.28.244.189@KRBTEST.COM (short username kudu)
I20260929 20:34:58.426838 20196 webserver.cc:460] Webserver started at http://127.28.244.189:43271/ using document root <none> and password file <none>
I20260929 20:34:58.427542 20196 fs_manager.cc:359] Metadata directory not provided
I20260929 20:34:58.427631 20196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:34:58.427992 20196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:34:58.430943 20196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-1/data/instance:
uuid: "ad0880fdd1924c4087fb41a41efb4360"
format_stamp: "Formatted at 2026-09-29 20:34:58 on dist-test-slave-p29l"
server_key: "d07e94b7e6ba73eb9c763a7a2f275398"
server_key_iv: "40fc71b541216d6f692d6d2b40dce8a4"
server_key_version: "encryptionkey@0"
I20260929 20:34:58.431767 20196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-1/wal/instance:
uuid: "ad0880fdd1924c4087fb41a41efb4360"
format_stamp: "Formatted at 2026-09-29 20:34:58 on dist-test-slave-p29l"
server_key: "d07e94b7e6ba73eb9c763a7a2f275398"
server_key_iv: "40fc71b541216d6f692d6d2b40dce8a4"
server_key_version: "encryptionkey@0"
I20260929 20:34:58.437170 20196 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 20:34:58.440899 20211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:58.442394 20196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260929 20:34:58.442603 20196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-1/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-1/wal
uuid: "ad0880fdd1924c4087fb41a41efb4360"
format_stamp: "Formatted at 2026-09-29 20:34:58 on dist-test-slave-p29l"
server_key: "d07e94b7e6ba73eb9c763a7a2f275398"
server_key_iv: "40fc71b541216d6f692d6d2b40dce8a4"
server_key_version: "encryptionkey@0"
I20260929 20:34:58.442834 20196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:34:58.468412 20196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:34:58.472560 20196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:34:58.472858 20196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:34:58.491657 20196 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.189:38609
I20260929 20:34:58.491686 20263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.189:38609 every 8 connection(s)
I20260929 20:34:58.493076 20196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-1/data/info.pb
I20260929 20:34:58.496999 20264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:34:58.502962 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 20196
I20260929 20:34:58.503163 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-1/wal/instance
I20260929 20:34:58.503712 29650 external_mini_cluster.cc:1428] Setting key fa54be9dcc9059c1b65c1050050d79b2
I20260929 20:34:58.504101 20264 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.244.190" port: 46381 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:34:58 dist-test-slave-p29l krb5kdc[20109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714098, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.189@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:34:58.530593 20266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:58.507729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40086 (local address 127.28.244.190:46381)
0929 20:34:58.508319 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:58.508327 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:58.509363 (+ 1036us) server_negotiation.cc:407] Connection header received
0929 20:34:58.512401 (+ 3038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:58.512406 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:58.512480 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:58.512571 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:58.514865 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:58.516167 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:58.517776 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:58.518547 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:58.518689 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:58.522925 (+ 4236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:58.522934 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:58.522942 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:58.522995 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:58.526915 (+ 3920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:58.527758 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:58.527767 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:58.527774 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:58.527965 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:58.528396 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:58.528403 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:58.528409 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:58.528823 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:58.528989 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:58.529802 (+ 813us) server_negotiation.cc:299] Negotiation successful
0929 20:34:58.530070 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":286,"threads_started":1}
I20260929 20:34:58.531256 20267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:58.508349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:46381 (local address 127.0.0.1:40086)
0929 20:34:58.509156 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:58.509211 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:58.512049 (+ 2838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:58.512882 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:58.512896 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:58.513751 (+ 855us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:34:58.514605 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:58.514637 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:58.516353 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:58.516357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:58.517612 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:58.517626 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:58.518767 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:58.518772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:58.519008 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:58.519767 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:34:58.519794 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:34:58.522633 (+ 2839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:34:58.527088 (+ 4455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:58.527096 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:58.527111 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:58.527620 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:58.528085 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:58.528089 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:58.528091 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:58.528276 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:58.528989 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:34:58.529001 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:34:58.529407 (+ 406us) client_negotiation.cc:769] Sending connection context
0929 20:34:58.529654 (+ 247us) client_negotiation.cc:240] Negotiation successful
0929 20:34:58.530061 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":295,"threads_started":1}
WARNING: no policy specified for kudu/127.28.244.188@KRBTEST.COM; defaulting to no policy
I20260929 20:34:58.533140 20264 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.244.189" port: 38609 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.244.188@KRBTEST.COM" created.
Sep 29 20:34:58 dist-test-slave-p29l krb5kdc[20109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714098, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.189@KRBTEST.COM for kudu/127.28.244.189@KRBTEST.COM
I20260929 20:34:58.549137 20267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:58.533882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:38609 (local address 127.0.0.1:40914)
0929 20:34:58.534100 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:58.534121 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:58.534237 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:58.535290 (+ 1053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:58.535297 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:58.535786 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:34:58.536005 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:58.536014 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:58.537404 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:58.537412 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:58.538276 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:58.538288 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:58.539165 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:58.539170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:58.539379 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:58.540082 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:34:58.540100 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:34:58.542132 (+ 2032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:34:58.546443 (+ 4311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:58.546448 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:58.546452 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:58.546928 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:58.547372 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:58.547376 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:58.547378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:58.547518 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:58.548319 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:34:58.548322 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:34:58.548436 (+ 114us) client_negotiation.cc:769] Sending connection context
0929 20:34:58.548555 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 20:34:58.548762 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":3}
I20260929 20:34:58.549573 20269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:58.534179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40914 (local address 127.28.244.189:38609)
0929 20:34:58.534783 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:58.534789 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:58.534814 (+ 25us) server_negotiation.cc:407] Connection header received
0929 20:34:58.534858 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:58.534863 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:58.535136 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:58.535309 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:58.536181 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:58.537263 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:58.538418 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:58.538986 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:58.539155 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:58.542348 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:58.542361 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:58.542376 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:58.542408 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:58.546293 (+ 3885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:58.547090 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:58.547102 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:58.547108 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:58.547253 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:58.547629 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:58.547637 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:58.547643 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:58.548185 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:58.548389 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:58.548743 (+ 354us) server_negotiation.cc:299] Negotiation successful
0929 20:34:58.549026 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":462,"thread_start_us":333,"threads_started":1}
I20260929 20:34:58.552088 20264 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.244.188" port: 37651 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:34:58.553264 20267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:58.552787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:37651 (local address 127.0.0.1:53194)
0929 20:34:58.553004 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:58.553110 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:37651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 20:34:58.553568 20214 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.244.188:37651: connect: Connection refused (error 111)
I20260929 20:34:58.555101 20267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:58.554762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:37651 (local address 127.0.0.1:53202)
0929 20:34:58.554920 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:58.554998 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:37651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 20:34:58.555421 20264 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.188:37651: Network error: Client connection negotiation failed: client connection to 127.28.244.188:37651: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.244.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.188.keytab.
Entry for principal kudu/127.28.244.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.188.keytab.
I20260929 20:34:58.569409 29650 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNrwVME/build/asan/bin/kudu
/tmp/dist-test-taskNrwVME/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.188:37651
--webserver_interface=127.28.244.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.188.keytab
--principal=kudu/127.28.244.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.244.148:39919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.244.190:46381,127.28.244.189:38609,127.28.244.188:37651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:34:58.604755 20264 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.188" port: 37651 } attempt: 1
I20260929 20:34:58.606007 20267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:58.605550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:37651 (local address 127.0.0.1:53204)
0929 20:34:58.605747 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:58.605854 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:37651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 20:34:58.607064 20267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:58.606737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:37651 (local address 127.0.0.1:53206)
0929 20:34:58.606877 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:58.606946 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:37651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 20:34:58.607332 20264 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.188:37651: Network error: Client connection negotiation failed: client connection to 127.28.244.188:37651: connect: Connection refused (error 111)
I20260929 20:34:58.669456 20189 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.189" port: 38609 } attempt: 5
I20260929 20:34:58.675571 20264 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.188" port: 37651 } attempt: 2
I20260929 20:34:58.676920 20267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:58.676348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:37651 (local address 127.0.0.1:53220)
0929 20:34:58.676598 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:58.676714 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:37651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260929 20:34:58.678187 20267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:58.677737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:37651 (local address 127.0.0.1:53234)
0929 20:34:58.677936 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:58.678040 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:37651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260929 20:34:58.678522 20264 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.188:37651: Network error: Client connection negotiation failed: client connection to 127.28.244.188:37651: connect: Connection refused (error 111)
Sep 29 20:34:58 dist-test-slave-p29l krb5kdc[20109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714097, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for kudu/127.28.244.189@KRBTEST.COM
I20260929 20:34:58.685662 20269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:58.670492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40926 (local address 127.28.244.189:38609)
0929 20:34:58.670695 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:58.670700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:58.670723 (+ 23us) server_negotiation.cc:407] Connection header received
0929 20:34:58.670780 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:58.670785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:58.670848 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:58.670935 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:58.672019 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:58.672732 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:58.674729 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:58.675200 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:58.675311 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:58.679701 (+ 4390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:58.679711 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:58.679722 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:58.679757 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:58.682675 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:58.683446 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:58.683453 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:58.683457 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:58.683599 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:58.684094 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:58.684100 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:58.684104 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:58.684438 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:58.684612 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:58.685062 (+ 450us) server_negotiation.cc:299] Negotiation successful
0929 20:34:58.685299 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:34:58.685683 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:58.670263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:38609 (local address 127.0.0.1:40926)
0929 20:34:58.670508 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:58.670531 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:58.670669 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:58.671033 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:58.671040 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:58.671569 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:34:58.671858 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:58.671874 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:58.672923 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:58.672931 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:58.674511 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:58.674529 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:58.675385 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:58.675391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:58.675621 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:58.676508 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:34:58.676537 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:34:58.679482 (+ 2945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:34:58.682844 (+ 3362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:58.682850 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:58.682855 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:58.683284 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:58.683729 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:58.683735 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:58.683739 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:58.683967 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:58.684574 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:34:58.684579 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:34:58.684759 (+ 180us) client_negotiation.cc:769] Sending connection context
0929 20:34:58.684929 (+ 170us) client_negotiation.cc:240] Negotiation successful
0929 20:34:58.685291 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260929 20:34:58.686973 20189 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.244.188" port: 37651 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:34:58.688191 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:58.687752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:37651 (local address 127.0.0.1:53240)
0929 20:34:58.687955 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:58.688050 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:37651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 20:34:58.689311 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:58.688827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:37651 (local address 127.0.0.1:53250)
0929 20:34:58.689010 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:58.689113 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:37651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 20:34:58.689582 20189 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.188:37651: Network error: Client connection negotiation failed: client connection to 127.28.244.188:37651: connect: Connection refused (error 111)
I20260929 20:34:58.740840 20189 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.188" port: 37651 } attempt: 1
I20260929 20:34:58.742022 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:58.741537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:37651 (local address 127.0.0.1:53262)
0929 20:34:58.741747 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:58.741851 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:37651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 20:34:58.743202 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:58.742816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:37651 (local address 127.0.0.1:53272)
0929 20:34:58.743016 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:58.743096 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:37651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260929 20:34:58.743422 20189 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.188:37651: Network error: Client connection negotiation failed: client connection to 127.28.244.188:37651: connect: Connection refused (error 111)
I20260929 20:34:58.769810 20264 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.188" port: 37651 } attempt: 3
I20260929 20:34:58.771013 20267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:58.770510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:37651 (local address 127.0.0.1:53282)
0929 20:34:58.770717 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:58.770813 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:37651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 20:34:58.772289 20267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:58.771759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:37651 (local address 127.0.0.1:53290)
0929 20:34:58.772094 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:58.772173 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:37651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221}
W20260929 20:34:58.772495 20264 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.188:37651: Network error: Client connection negotiation failed: client connection to 127.28.244.188:37651: connect: Connection refused (error 111)
I20260929 20:34:58.811651 20189 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.188" port: 37651 } attempt: 2
I20260929 20:34:58.812968 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:58.812455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:37651 (local address 127.0.0.1:53302)
0929 20:34:58.812700 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:58.812786 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:37651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 20:34:58.814095 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:58.813703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:37651 (local address 127.0.0.1:53304)
0929 20:34:58.813902 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:58.813985 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:37651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 20:34:58.814284 20189 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.188:37651: Network error: Client connection negotiation failed: client connection to 127.28.244.188:37651: connect: Connection refused (error 111)
I20260929 20:34:58.915727 20264 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.188" port: 37651 } attempt: 4
I20260929 20:34:58.917048 20267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:58.916474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:37651 (local address 127.0.0.1:53318)
0929 20:34:58.916743 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:58.916851 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:37651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 20:34:58.918109 20267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:58.917751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:37651 (local address 127.0.0.1:53326)
0929 20:34:58.917900 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:58.917982 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:37651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 20:34:58.918383 20264 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.188:37651: Network error: Client connection negotiation failed: client connection to 127.28.244.188:37651: connect: Connection refused (error 111)
W20260929 20:34:58.918908 20272 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 20:34:58.919255 20272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:34:58.919364 20272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:34:58.919497 20272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 20:34:58.920493 20189 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.188" port: 37651 } attempt: 3
I20260929 20:34:58.921584 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:58.921071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:37651 (local address 127.0.0.1:53342)
0929 20:34:58.921281 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:58.921380 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:37651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 20:34:58.922886 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:58.922363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:37651 (local address 127.0.0.1:53348)
0929 20:34:58.922632 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:58.922731 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.244.188:37651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
W20260929 20:34:58.923092 20189 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.244.188:37651: Network error: Client connection negotiation failed: client connection to 127.28.244.188:37651: connect: Connection refused (error 111)
W20260929 20:34:58.927914 20272 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:34:58.928006 20272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:34:58.928131 20272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:34:58.928206 20272 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:34:58.928292 20272 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:34:58.938898 20272 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.244.148:39919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.244.190:46381,127.28.244.189:38609,127.28.244.188:37651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/krb5kdc/kudu_127.28.244.188.keytab
--principal=kudu/127.28.244.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.244.188:37651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.244.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:34:58.940374 20272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:34:58.941704 20272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:34:58.954077 20277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:58.954007 20278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:34:58.954119 20280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:34:58.955127 20272 server_base.cc:1034] running on GCE node
I20260929 20:34:58.956022 20272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:34:58.957484 20272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:34:58.958768 20272 hybrid_clock.cc:648] HybridClock initialized: now 1790714098958703 us; error 69 us; skew 500 ppm
I20260929 20:34:58.959117 20272 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:34:58 dist-test-slave-p29l krb5kdc[20109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714098, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:34:58.962666 20272 init.cc:377] Logged in from keytab as kudu/127.28.244.188@KRBTEST.COM (short username kudu)
I20260929 20:34:58.969029 20272 webserver.cc:460] Webserver started at http://127.28.244.188:34923/ using document root <none> and password file <none>
I20260929 20:34:58.969787 20272 fs_manager.cc:359] Metadata directory not provided
I20260929 20:34:58.969910 20272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:34:58.970248 20272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:34:58.973308 20272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-2/data/instance:
uuid: "f70b427d6a734f019f7793b035931ae7"
format_stamp: "Formatted at 2026-09-29 20:34:58 on dist-test-slave-p29l"
server_key: "31f6a493f9b9e4272f9e053c389ee687"
server_key_iv: "15d8490579cd72fc7a2e1d951a578f5e"
server_key_version: "encryptionkey@0"
I20260929 20:34:58.974133 20272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-2/wal/instance:
uuid: "f70b427d6a734f019f7793b035931ae7"
format_stamp: "Formatted at 2026-09-29 20:34:58 on dist-test-slave-p29l"
server_key: "31f6a493f9b9e4272f9e053c389ee687"
server_key_iv: "15d8490579cd72fc7a2e1d951a578f5e"
server_key_version: "encryptionkey@0"
I20260929 20:34:58.979712 20272 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 20:34:58.983225 20287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:34:58.984632 20272 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 20:34:58.984824 20272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-2/data,/tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-2/wal
uuid: "f70b427d6a734f019f7793b035931ae7"
format_stamp: "Formatted at 2026-09-29 20:34:58 on dist-test-slave-p29l"
server_key: "31f6a493f9b9e4272f9e053c389ee687"
server_key_iv: "15d8490579cd72fc7a2e1d951a578f5e"
server_key_version: "encryptionkey@0"
I20260929 20:34:58.985062 20272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:34:59.007097 20272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:34:59.011745 20272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:34:59.012069 20272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:34:59.031457 20272 rpc_server.cc:307] RPC server started. Bound to: 127.28.244.188:37651
I20260929 20:34:59.031481 20338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.244.188:37651 every 8 connection(s)
I20260929 20:34:59.032963 20272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-2/data/info.pb
I20260929 20:34:59.036624 29650 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNrwVME/build/asan/bin/kudu as pid 20272
I20260929 20:34:59.036813 29650 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNrwVME/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713721201454-29650-0/minicluster-data/master-2/wal/instance
I20260929 20:34:59.036904 20339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:34:59.037365 29650 external_mini_cluster.cc:1428] Setting key 1bdc8eb9d393ce0d05b42f1612b4ccad
I20260929 20:34:59.046700 20339 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.244.190" port: 46381 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:34:59 dist-test-slave-p29l krb5kdc[20109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.188@KRBTEST.COM
Sep 29 20:34:59 dist-test-slave-p29l krb5kdc[20109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714098, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.188@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
I20260929 20:34:59.061301 20342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:59.039099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53364 (local address 127.28.244.188:37651)
0929 20:34:59.039958 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:59.039972 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:59.040011 (+ 39us) server_negotiation.cc:407] Connection header received
0929 20:34:59.041757 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:59.041779 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:59.042164 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:59.042485 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:59.044289 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:59.045591 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:59.047466 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:59.048261 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:59.048394 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:59.053099 (+ 4705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:59.053109 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:59.053125 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:59.053177 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:59.056903 (+ 3726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:59.057827 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:59.057838 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:59.057851 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:59.058069 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:59.058454 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:59.058460 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:59.058463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:59.059053 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:59.059361 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:59.059840 (+ 479us) server_negotiation.cc:299] Negotiation successful
0929 20:34:59.060282 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":404,"threads_started":1}
I20260929 20:34:59.066375 20343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:59.049657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40096 (local address 127.28.244.190:46381)
0929 20:34:59.050169 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:59.050177 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:59.050620 (+ 443us) server_negotiation.cc:407] Connection header received
0929 20:34:59.050842 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:59.050847 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:59.050919 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:59.051031 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:59.052295 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:59.053118 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:59.054687 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:59.055364 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:59.055497 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:59.059222 (+ 3725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:59.059227 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:59.059230 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:59.059271 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:59.063126 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:59.064041 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:59.064059 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:59.064063 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:59.064279 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:59.064758 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:59.064770 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:59.064773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:59.065064 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:59.065246 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:59.065710 (+ 464us) server_negotiation.cc:299] Negotiation successful
0929 20:34:59.065927 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":272,"threads_started":1}
I20260929 20:34:59.066375 20344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:59.049671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.190:46381 (local address 127.0.0.1:40096)
0929 20:34:59.050470 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:59.050508 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:59.050702 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:59.051092 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:59.051100 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:59.051740 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:34:59.052125 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:59.052140 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:59.053300 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:59.053305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:59.054513 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:59.054526 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:59.055567 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:59.055572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:59.055718 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:59.056424 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:34:59.056442 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:34:59.059021 (+ 2579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:34:59.063272 (+ 4251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:59.063283 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:59.063291 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:59.063896 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:59.064417 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:59.064425 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:59.064432 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:59.064634 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:59.065196 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:34:59.065203 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:34:59.065410 (+ 207us) client_negotiation.cc:769] Sending connection context
0929 20:34:59.065597 (+ 187us) client_negotiation.cc:240] Negotiation successful
0929 20:34:59.065927 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":467,"threads_started":1}
I20260929 20:34:59.067904 20339 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.244.189" port: 38609 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:34:59 dist-test-slave-p29l krb5kdc[20109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.190@KRBTEST.COM
Sep 29 20:34:59 dist-test-slave-p29l krb5kdc[20109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714098, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.188@KRBTEST.COM for kudu/127.28.244.189@KRBTEST.COM
I20260929 20:34:59.081328 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:59.064412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40104 (local address 127.28.244.190:46381)
0929 20:34:59.065037 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:59.065041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:59.065065 (+ 24us) server_negotiation.cc:407] Connection header received
0929 20:34:59.065105 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:59.065108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:59.065162 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:59.065281 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:59.066534 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:59.067507 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:59.069007 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:59.069663 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:59.069799 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:59.074819 (+ 5020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:59.074825 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:59.074828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:59.074861 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:59.078159 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:59.078939 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:59.078946 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:59.078950 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:59.079121 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:59.079589 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:59.079595 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:59.079598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:59.079999 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:59.080204 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:59.080634 (+ 430us) server_negotiation.cc:299] Negotiation successful
0929 20:34:59.080878 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":332,"threads_started":1}
I20260929 20:34:59.082561 20344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:59.068590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.189:38609 (local address 127.0.0.1:40938)
0929 20:34:59.068750 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:59.068770 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:59.068868 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:59.069196 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:59.069201 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:59.069670 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:34:59.069862 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:59.069876 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:59.071212 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:59.071216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:59.072157 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:59.072169 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:59.073150 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:59.073156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:59.073361 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:59.073918 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:34:59.073943 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:34:59.076281 (+ 2338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:34:59.079922 (+ 3641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:59.079926 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:59.079929 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:59.080476 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:59.080882 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:59.080886 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:59.080889 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:59.081077 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:59.081692 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:34:59.081696 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:34:59.081870 (+ 174us) client_negotiation.cc:769] Sending connection context
0929 20:34:59.082015 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 20:34:59.082227 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 20:34:59.082780 20269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:59.068696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40938 (local address 127.28.244.189:38609)
0929 20:34:59.068886 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:59.068893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:59.068913 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:34:59.068976 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:59.068984 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:59.069069 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:59.069144 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:59.070023 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:59.071060 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:59.072298 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:59.072994 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:59.073092 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:59.076510 (+ 3418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:59.076517 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:59.076520 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:59.076554 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:59.079728 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:59.080599 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:59.080605 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:59.080608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:59.080754 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:59.081197 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:59.081203 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:59.081207 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:59.081565 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:59.081708 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:59.082149 (+ 441us) server_negotiation.cc:299] Negotiation successful
0929 20:34:59.082377 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 20:34:59.084003 20339 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.244.188" port: 37651 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:34:59 dist-test-slave-p29l krb5kdc[20109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714098, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.188@KRBTEST.COM for kudu/127.28.244.188@KRBTEST.COM
Sep 29 20:34:59 dist-test-slave-p29l krb5kdc[20109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.244.189@KRBTEST.COM
I20260929 20:34:59.098424 20342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:59.084952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53372 (local address 127.28.244.188:37651)
0929 20:34:59.085128 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:59.085136 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:59.085155 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:34:59.085205 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:59.085210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:59.085279 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:59.085355 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:59.086259 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:59.087283 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:59.088878 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:59.089550 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:59.089689 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:59.093081 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:59.093086 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:59.093089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:59.093117 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:59.095714 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:59.096391 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:59.096399 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:59.096405 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:59.096538 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:59.096899 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:59.096903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:59.096907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:59.097227 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:59.097381 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:59.097753 (+ 372us) server_negotiation.cc:299] Negotiation successful
0929 20:34:59.098029 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":62}
I20260929 20:34:59.098510 20344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:59.084720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:37651 (local address 127.0.0.1:53372)
0929 20:34:59.084867 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:59.084885 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:59.084978 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:59.085415 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:59.085419 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:59.085948 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:34:59.086123 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:59.086136 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:59.087418 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:59.087422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:59.088697 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:59.088710 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:59.089703 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:59.089707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:59.089924 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:59.090549 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:34:59.090566 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:34:59.092899 (+ 2333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:34:59.095886 (+ 2987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:59.095894 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:59.095898 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:59.096272 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:59.096645 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:59.096651 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:59.096654 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:59.096788 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:59.097342 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:34:59.097346 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:34:59.097498 (+ 152us) client_negotiation.cc:769] Sending connection context
0929 20:34:59.097663 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 20:34:59.098169 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":205}
I20260929 20:34:59.099207 20269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:59.083424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40940 (local address 127.28.244.189:38609)
0929 20:34:59.083589 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:59.083594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:59.083608 (+ 14us) server_negotiation.cc:407] Connection header received
0929 20:34:59.083646 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:59.083650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:59.083701 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:59.083837 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:59.085221 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:59.086347 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:59.087939 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:59.088592 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:59.088711 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:59.093746 (+ 5035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:59.093754 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:59.093760 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:59.093804 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:59.096517 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:59.097235 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:59.097240 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:59.097242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:59.097397 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:59.097805 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:59.097809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:59.097811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:59.098070 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:59.098349 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:59.098629 (+ 280us) server_negotiation.cc:299] Negotiation successful
0929 20:34:59.098851 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:34:59.104095 29650 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:34:59.104691 20339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f70b427d6a734f019f7793b035931ae7: Bootstrap starting.
I20260929 20:34:59.108628 20339 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f70b427d6a734f019f7793b035931ae7: Neither blocks nor log segments found. Creating new log.
I20260929 20:34:59.108846 20189 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.188" port: 37651 } attempt: 4
I20260929 20:34:59.110014 20339 log.cc:824] T 00000000000000000000000000000000 P f70b427d6a734f019f7793b035931ae7: Log is configured to *not* fsync() on all Append() calls
I20260929 20:34:59.106576 20164 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c6c8af3248b54a3c85fc56aa28854284"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40104
I20260929 20:34:59.113597 20339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f70b427d6a734f019f7793b035931ae7: No bootstrap required, opened a new log
Sep 29 20:34:59 dist-test-slave-p29l krb5kdc[20109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714097, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.190@KRBTEST.COM for kudu/127.28.244.188@KRBTEST.COM
I20260929 20:34:59.123996 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:59.109554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:37651 (local address 127.0.0.1:53374)
0929 20:34:59.109784 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:59.109800 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:59.109901 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:59.110441 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:59.110446 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:59.111055 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:34:59.111292 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:59.111309 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:59.112865 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:59.112869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:59.113906 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:59.113919 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:59.114759 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:59.114765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:59.114964 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:59.115620 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:34:59.115647 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:34:59.117814 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:34:59.121317 (+ 3503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:59.121325 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:59.121330 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:59.121722 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:59.122122 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:59.122127 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:59.122131 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:59.122295 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:59.122902 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:34:59.122908 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:34:59.123098 (+ 190us) client_negotiation.cc:769] Sending connection context
0929 20:34:59.123256 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 20:34:59.123563 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260929 20:34:59.124049 20342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:59.109578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53374 (local address 127.28.244.188:37651)
0929 20:34:59.110116 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:59.110121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:59.110161 (+ 40us) server_negotiation.cc:407] Connection header received
0929 20:34:59.110218 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:59.110222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:59.110295 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:59.110409 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:59.111469 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:59.112678 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:59.114087 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:59.114586 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:59.114726 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:59.118007 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:59.118014 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:59.118018 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:59.118059 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:59.121163 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:59.121853 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:59.121861 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:59.121869 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:59.122005 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:59.122410 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:59.122415 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:59.122418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:59.122741 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:59.122956 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:59.123414 (+ 458us) server_negotiation.cc:299] Negotiation successful
0929 20:34:59.123638 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413}
I20260929 20:34:59.125408 20339 raft_consensus.cc:348] T 00000000000000000000000000000000 P f70b427d6a734f019f7793b035931ae7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c8af3248b54a3c85fc56aa28854284" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 46381 } } peers { permanent_uuid: "ad0880fdd1924c4087fb41a41efb4360" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 38609 } } peers { permanent_uuid: "f70b427d6a734f019f7793b035931ae7" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 37651 } }
I20260929 20:34:59.125903 20339 raft_consensus.cc:729] T 00000000000000000000000000000000 P f70b427d6a734f019f7793b035931ae7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f70b427d6a734f019f7793b035931ae7, State: Initialized, Role: FOLLOWER
I20260929 20:34:59.126380 20339 consensus_queue.cc:260] T 00000000000000000000000000000000 P f70b427d6a734f019f7793b035931ae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c8af3248b54a3c85fc56aa28854284" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 46381 } } peers { permanent_uuid: "ad0880fdd1924c4087fb41a41efb4360" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 38609 } } peers { permanent_uuid: "f70b427d6a734f019f7793b035931ae7" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 37651 } }
I20260929 20:34:59.127386 20347 sys_catalog.cc:455] T 00000000000000000000000000000000 P f70b427d6a734f019f7793b035931ae7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c8af3248b54a3c85fc56aa28854284" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 46381 } } peers { permanent_uuid: "ad0880fdd1924c4087fb41a41efb4360" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 38609 } } peers { permanent_uuid: "f70b427d6a734f019f7793b035931ae7" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 37651 } } }
I20260929 20:34:59.127767 20347 sys_catalog.cc:458] T 00000000000000000000000000000000 P f70b427d6a734f019f7793b035931ae7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:34:59.128638 20339 sys_catalog.cc:564] T 00000000000000000000000000000000 P f70b427d6a734f019f7793b035931ae7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:34:59.129978 20189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6c8af3248b54a3c85fc56aa28854284: Bootstrap starting.
I20260929 20:34:59.133998 20189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6c8af3248b54a3c85fc56aa28854284: Neither blocks nor log segments found. Creating new log.
I20260929 20:34:59.135289 20189 log.cc:824] T 00000000000000000000000000000000 P c6c8af3248b54a3c85fc56aa28854284: Log is configured to *not* fsync() on all Append() calls
W20260929 20:34:59.139061 20358 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f70b427d6a734f019f7793b035931ae7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 20:34:59.139196 20358 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 20:34:59.149379 20189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6c8af3248b54a3c85fc56aa28854284: No bootstrap required, opened a new log
I20260929 20:34:59.136377 20272 master_runner.cc:186] Error getting master registration for 127.28.244.190:46381: OK, instance_id { permanent_uuid: "c6c8af3248b54a3c85fc56aa28854284" instance_seqno: 1790714097956746 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 20:34:59.152540 20272 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 20:34:59.155530 20189 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6c8af3248b54a3c85fc56aa28854284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c8af3248b54a3c85fc56aa28854284" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 46381 } } peers { permanent_uuid: "ad0880fdd1924c4087fb41a41efb4360" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 38609 } } peers { permanent_uuid: "f70b427d6a734f019f7793b035931ae7" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 37651 } }
I20260929 20:34:59.155916 20189 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6c8af3248b54a3c85fc56aa28854284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6c8af3248b54a3c85fc56aa28854284, State: Initialized, Role: FOLLOWER
I20260929 20:34:59.156383 20189 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6c8af3248b54a3c85fc56aa28854284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c8af3248b54a3c85fc56aa28854284" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 46381 } } peers { permanent_uuid: "ad0880fdd1924c4087fb41a41efb4360" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 38609 } } peers { permanent_uuid: "f70b427d6a734f019f7793b035931ae7" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 37651 } }
I20260929 20:34:59.157404 20361 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6c8af3248b54a3c85fc56aa28854284 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c8af3248b54a3c85fc56aa28854284" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 46381 } } peers { permanent_uuid: "ad0880fdd1924c4087fb41a41efb4360" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 38609 } } peers { permanent_uuid: "f70b427d6a734f019f7793b035931ae7" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 37651 } } }
I20260929 20:34:59.157869 20361 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6c8af3248b54a3c85fc56aa28854284 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:34:59.158697 20189 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6c8af3248b54a3c85fc56aa28854284 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:34:59.158741 20164 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6c8af3248b54a3c85fc56aa28854284 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 20:34:59.158938 20164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6c8af3248b54a3c85fc56aa28854284 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:34:59.163146 20164 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6c8af3248b54a3c85fc56aa28854284 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c8af3248b54a3c85fc56aa28854284" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 46381 } } peers { permanent_uuid: "ad0880fdd1924c4087fb41a41efb4360" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 38609 } } peers { permanent_uuid: "f70b427d6a734f019f7793b035931ae7" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 37651 } }
I20260929 20:34:59.166688 20164 leader_election.cc:288] T 00000000000000000000000000000000 P c6c8af3248b54a3c85fc56aa28854284 [CANDIDATE]: Term 1 election: Requested vote from peers ad0880fdd1924c4087fb41a41efb4360 (127.28.244.189:38609), f70b427d6a734f019f7793b035931ae7 (127.28.244.188:37651)
I20260929 20:34:59.167382 20314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6c8af3248b54a3c85fc56aa28854284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f70b427d6a734f019f7793b035931ae7"
I20260929 20:34:59.168045 20314 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f70b427d6a734f019f7793b035931ae7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:34:59.172709 20314 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f70b427d6a734f019f7793b035931ae7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6c8af3248b54a3c85fc56aa28854284 in term 1.
I20260929 20:34:59.173671 20140 leader_election.cc:302] T 00000000000000000000000000000000 P c6c8af3248b54a3c85fc56aa28854284 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6c8af3248b54a3c85fc56aa28854284, f70b427d6a734f019f7793b035931ae7; no voters:
I20260929 20:34:59.174356 20361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6c8af3248b54a3c85fc56aa28854284 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:34:59.175838 20372 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 20:34:59.185590 20361 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6c8af3248b54a3c85fc56aa28854284 [term 1 LEADER]: Becoming Leader. State: Replica: c6c8af3248b54a3c85fc56aa28854284, State: Running, Role: LEADER
I20260929 20:34:59.186298 20361 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6c8af3248b54a3c85fc56aa28854284 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c8af3248b54a3c85fc56aa28854284" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 46381 } } peers { permanent_uuid: "ad0880fdd1924c4087fb41a41efb4360" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 38609 } } peers { permanent_uuid: "f70b427d6a734f019f7793b035931ae7" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 37651 } }
W20260929 20:34:59.196024 20143 tablet.cc:2369] T 00000000000000000000000000000000 P c6c8af3248b54a3c85fc56aa28854284: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:34:59.202591 20374 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6c8af3248b54a3c85fc56aa28854284 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6c8af3248b54a3c85fc56aa28854284. Latest consensus state: current_term: 1 leader_uuid: "c6c8af3248b54a3c85fc56aa28854284" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c8af3248b54a3c85fc56aa28854284" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 46381 } } peers { permanent_uuid: "ad0880fdd1924c4087fb41a41efb4360" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 38609 } } peers { permanent_uuid: "f70b427d6a734f019f7793b035931ae7" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 37651 } } }
I20260929 20:34:59.203245 20374 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6c8af3248b54a3c85fc56aa28854284 [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:59.193297 20122 master_runner.cc:186] Error getting master registration for 127.28.244.189:38609: OK, instance_id { permanent_uuid: "ad0880fdd1924c4087fb41a41efb4360" instance_seqno: 1790714098486389 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 20:34:59.204134 20122 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 20:34:59.204838 20380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:34:59.217619 20264 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.244.188" port: 37651 } attempt: 5
I20260929 20:34:59.220368 20380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:34:59.248039 20314 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f70b427d6a734f019f7793b035931ae7 [term 1 FOLLOWER]: Refusing update from remote peer c6c8af3248b54a3c85fc56aa28854284: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:34:59.249228 20374 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c6c8af3248b54a3c85fc56aa28854284 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f70b427d6a734f019f7793b035931ae7" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Sep 29 20:34:59 dist-test-slave-p29l krb5kdc[20109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714098, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.244.189@KRBTEST.COM for kudu/127.28.244.188@KRBTEST.COM
W20260929 20:34:59.264343 20293 tablet.cc:2369] T 00000000000000000000000000000000 P f70b427d6a734f019f7793b035931ae7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:34:59.279088 20342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:59.215342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53386 (local address 127.28.244.188:37651)
0929 20:34:59.215584 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:59.215591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:59.234268 (+ 18677us) server_negotiation.cc:407] Connection header received
0929 20:34:59.234586 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:59.234595 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:59.234696 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:59.234777 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:59.237140 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:59.238218 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:59.242807 (+ 4589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:59.243495 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:59.243628 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:59.264171 (+ 20543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:59.264179 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:59.264186 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:59.264246 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:59.268702 (+ 4456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:59.271312 (+ 2610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:59.271346 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:59.271349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:59.271685 (+ 336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:59.272233 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:59.272244 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:59.272249 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:59.272682 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:59.278208 (+ 5526us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:59.278390 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 20:34:59.278655 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 20:34:59.280221 20347 sys_catalog.cc:455] T 00000000000000000000000000000000 P f70b427d6a734f019f7793b035931ae7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6c8af3248b54a3c85fc56aa28854284. Latest consensus state: current_term: 1 leader_uuid: "c6c8af3248b54a3c85fc56aa28854284" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c8af3248b54a3c85fc56aa28854284" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 46381 } } peers { permanent_uuid: "ad0880fdd1924c4087fb41a41efb4360" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 38609 } } peers { permanent_uuid: "f70b427d6a734f019f7793b035931ae7" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 37651 } } }
I20260929 20:34:59.280539 20347 sys_catalog.cc:458] T 00000000000000000000000000000000 P f70b427d6a734f019f7793b035931ae7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:34:59.284718 20385 mvcc.cc:205] Tried to move back new op lower bound from 7334764950472527872 to 7334764950272798720. Current Snapshot: MvccSnapshot[applied={T|T < 7334764950472527872}]
I20260929 20:34:59.286536 20387 mvcc.cc:205] Tried to move back new op lower bound from 7334764950472527872 to 7334764950272798720. Current Snapshot: MvccSnapshot[applied={T|T < 7334764950472527872}]
I20260929 20:34:59.297045 20361 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6c8af3248b54a3c85fc56aa28854284 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c6c8af3248b54a3c85fc56aa28854284" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c8af3248b54a3c85fc56aa28854284" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 46381 } } peers { permanent_uuid: "ad0880fdd1924c4087fb41a41efb4360" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 38609 } } peers { permanent_uuid: "f70b427d6a734f019f7793b035931ae7" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 37651 } } }
I20260929 20:34:59.297441 20361 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6c8af3248b54a3c85fc56aa28854284 [sys.catalog]: This master's current role is: LEADER
I20260929 20:34:59.298358 20380 catalog_manager.cc:1347] Generated new cluster ID: 2d74dce783554b82852d2a9eafd66cdc
I20260929 20:34:59.298457 20380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:34:59.303001 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:59.203950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40114 (local address 127.28.244.190:46381)
0929 20:34:59.204199 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:59.204205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:59.214426 (+ 10221us) server_negotiation.cc:407] Connection header received
0929 20:34:59.214695 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:59.214701 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:59.214793 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:59.214923 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:59.235644 (+ 20721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:59.236866 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:59.246122 (+ 9256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:59.246885 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:59.246982 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:59.285756 (+ 38774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:59.285767 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:59.285775 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:59.285833 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:59.290992 (+ 5159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:59.294314 (+ 3322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:59.294325 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:59.294330 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:59.294676 (+ 346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:59.297385 (+ 2709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:59.297393 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:59.297399 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:59.297861 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:59.298125 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:59.302035 (+ 3910us) server_negotiation.cc:299] Negotiation successful
0929 20:34:59.302461 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 20:34:59.304929 20269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:59.203552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40948 (local address 127.28.244.189:38609)
0929 20:34:59.203744 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:59.203749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:59.217782 (+ 14033us) server_negotiation.cc:407] Connection header received
0929 20:34:59.218791 (+ 1009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:59.218797 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:59.218880 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:59.218957 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:59.238777 (+ 19820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:59.239976 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:59.248545 (+ 8569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:59.249212 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:59.249317 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:59.272997 (+ 23680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:59.273005 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:59.273009 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:59.273069 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:59.277740 (+ 4671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:59.296316 (+ 18576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:59.296323 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:59.296326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:59.296539 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:59.299758 (+ 3219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:59.299765 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:59.299768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:59.300236 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:59.300484 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:59.304330 (+ 3846us) server_negotiation.cc:299] Negotiation successful
0929 20:34:59.304587 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:34:59.305604 20347 sys_catalog.cc:455] T 00000000000000000000000000000000 P f70b427d6a734f019f7793b035931ae7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c6c8af3248b54a3c85fc56aa28854284" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c8af3248b54a3c85fc56aa28854284" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 46381 } } peers { permanent_uuid: "ad0880fdd1924c4087fb41a41efb4360" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 38609 } } peers { permanent_uuid: "f70b427d6a734f019f7793b035931ae7" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 37651 } } }
I20260929 20:34:59.305902 20347 sys_catalog.cc:458] T 00000000000000000000000000000000 P f70b427d6a734f019f7793b035931ae7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:34:59.312789 20267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:59.218209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.244.188:37651 (local address 127.0.0.1:53388)
0929 20:34:59.218490 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 20:34:59.218517 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 20:34:59.218666 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:34:59.242816 (+ 24150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:34:59.242821 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:34:59.243398 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:34:59.243611 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:59.243623 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:59.252437 (+ 8814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:59.252442 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:59.253890 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:34:59.253908 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:34:59.259221 (+ 5313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:34:59.259232 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:34:59.259526 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:59.260379 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:34:59.260411 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:34:59.263652 (+ 3241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:34:59.309723 (+ 46071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:59.309729 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:59.309734 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:59.310301 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:59.310812 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:34:59.310817 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:34:59.310820 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:34:59.310992 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:34:59.311677 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:34:59.311684 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:34:59.311911 (+ 227us) client_negotiation.cc:769] Sending connection context
0929 20:34:59.312071 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 20:34:59.312393 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 20:34:59.318869 20384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:34:59.218568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53388 (local address 127.28.244.188:37651)
0929 20:34:59.242344 (+ 23776us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:59.242351 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:59.242373 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:34:59.242428 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:59.242434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:59.242521 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:59.243752 (+ 1231us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:59.244072 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:59.245363 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:59.254634 (+ 9271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:34:59.258748 (+ 4114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:34:59.264148 (+ 5400us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:34:59.269175 (+ 5027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:34:59.269187 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:34:59.269194 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:34:59.269358 (+ 164us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:34:59.309529 (+ 40171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:59.310438 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:59.310444 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:59.310448 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:59.310693 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:34:59.311123 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:34:59.311130 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:34:59.311134 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:34:59.311496 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:34:59.311666 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 20:34:59.313429 (+ 1763us) server_negotiation.cc:299] Negotiation successful
0929 20:34:59.313944 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23605,"thread_start_us":24037,"threads_started":1}
I20260929 20:34:59.329798 20264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad0880fdd1924c4087fb41a41efb4360: Bootstrap starting.
I20260929 20:34:59.332086 20380 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:34:59.334090 20380 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:34:59.339217 20264 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad0880fdd1924c4087fb41a41efb4360: Neither blocks nor log segments found. Creating new log.
I20260929 20:34:59.341864 20264 log.cc:824] T 00000000000000000000000000000000 P ad0880fdd1924c4087fb41a41efb4360: Log is configured to *not* fsync() on all Append() calls
I20260929 20:34:59.357499 20264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad0880fdd1924c4087fb41a41efb4360: No bootstrap required, opened a new log
I20260929 20:34:59.359695 20380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6c8af3248b54a3c85fc56aa28854284: Generated new TSK 0
I20260929 20:34:59.360961 20380 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:34:59.383925 20264 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad0880fdd1924c4087fb41a41efb4360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c8af3248b54a3c85fc56aa28854284" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 46381 } } peers { permanent_uuid: "ad0880fdd1924c4087fb41a41efb4360" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 38609 } } peers { permanent_uuid: "f70b427d6a734f019f7793b035931ae7" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 37651 } }
I20260929 20:34:59.385054 20264 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad0880fdd1924c4087fb41a41efb4360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad0880fdd1924c4087fb41a41efb4360, State: Initialized, Role: FOLLOWER
I20260929 20:34:59.386129 20264 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad0880fdd1924c4087fb41a41efb4360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c8af3248b54a3c85fc56aa28854284" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 46381 } } peers { permanent_uuid: "ad0880fdd1924c4087fb41a41efb4360" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 38609 } } peers { permanent_uuid: "f70b427d6a734f019f7793b035931ae7" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 37651 } }
I20260929 20:34:59.394750 20392 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad0880fdd1924c4087fb41a41efb4360 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c8af3248b54a3c85fc56aa28854284" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 46381 } } peers { permanent_uuid: "ad0880fdd1924c4087fb41a41efb4360" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 38609 } } peers { permanent_uuid: "f70b427d6a734f019f7793b035931ae7" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 37651 } } }
I20260929 20:34:59.395246 20392 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad0880fdd1924c4087fb41a41efb4360 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:34:59.397912 20238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6c8af3248b54a3c85fc56aa28854284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ad0880fdd1924c4087fb41a41efb4360"
I20260929 20:34:59.398411 20238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad0880fdd1924c4087fb41a41efb4360 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:34:59.402105 20238 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ad0880fdd1924c4087fb41a41efb4360 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6c8af3248b54a3c85fc56aa28854284 in term 1.
I20260929 20:34:59.403249 20237 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ad0880fdd1924c4087fb41a41efb4360 [term 1 FOLLOWER]: Refusing update from remote peer c6c8af3248b54a3c85fc56aa28854284: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:34:59.404004 20264 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad0880fdd1924c4087fb41a41efb4360 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:34:59.404181 20374 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c6c8af3248b54a3c85fc56aa28854284 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad0880fdd1924c4087fb41a41efb4360" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 38609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:34:59.468626 20392 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad0880fdd1924c4087fb41a41efb4360 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6c8af3248b54a3c85fc56aa28854284. Latest consensus state: current_term: 1 leader_uuid: "c6c8af3248b54a3c85fc56aa28854284" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c8af3248b54a3c85fc56aa28854284" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 46381 } } peers { permanent_uuid: "ad0880fdd1924c4087fb41a41efb4360" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 38609 } } peers { permanent_uuid: "f70b427d6a734f019f7793b035931ae7" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 37651 } } }
I20260929 20:34:59.469098 20392 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad0880fdd1924c4087fb41a41efb4360 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:34:59.468672 20400 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad0880fdd1924c4087fb41a41efb4360 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c6c8af3248b54a3c85fc56aa28854284" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c8af3248b54a3c85fc56aa28854284" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 46381 } } peers { permanent_uuid: "ad0880fdd1924c4087fb41a41efb4360" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 38609 } } peers { permanent_uuid: "f70b427d6a734f019f7793b035931ae7" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 37651 } } }
I20260929 20:34:59.469540 20400 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad0880fdd1924c4087fb41a41efb4360 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:34:59.502867 20417 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260929 20:34:59.507150 20269 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:34:59.494319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40958 (local address 127.28.244.189:38609)
0929 20:34:59.494619 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:59.494628 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:59.505170 (+ 10542us) server_negotiation.cc:407] Connection header received
0929 20:34:59.505512 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:59.505518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:59.505594 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:59.505662 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:59.506895 (+ 1233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40958: BlockingRecv error: recv got EOF from 127.0.0.1:40958 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260929 20:34:59.508104 20345 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:34:59.491731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40126 (local address 127.28.244.190:46381)
0929 20:34:59.492012 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:59.492021 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:59.505220 (+ 13199us) server_negotiation.cc:407] Connection header received
0929 20:34:59.505284 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:59.505290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:59.505365 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:59.505459 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:59.507831 (+ 2372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40126: BlockingRecv error: recv got EOF from 127.0.0.1:40126 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
W20260929 20:34:59.516773 20384 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:34:59.508643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53394 (local address 127.28.244.188:37651)
0929 20:34:59.508839 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:59.508845 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:59.515069 (+ 6224us) server_negotiation.cc:407] Connection header received
0929 20:34:59.515333 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:59.515340 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:59.515449 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:59.515530 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:59.516536 (+ 1006us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53394: BlockingRecv error: recv got EOF from 127.0.0.1:53394 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 20:34:59.538403 20269 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:34:59.529676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40966 (local address 127.28.244.189:38609)
0929 20:34:59.529873 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:59.529880 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:59.536606 (+ 6726us) server_negotiation.cc:407] Connection header received
0929 20:34:59.536875 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:59.536882 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:59.537018 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:59.537146 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:59.538110 (+ 964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40966: BlockingRecv error: recv got EOF from 127.0.0.1:40966 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 20:34:59.554260 20269 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:34:59.546123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40982 (local address 127.28.244.189:38609)
0929 20:34:59.546324 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:59.546331 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:59.552571 (+ 6240us) server_negotiation.cc:407] Connection header received
0929 20:34:59.552790 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:59.552797 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:59.552926 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:59.553058 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:59.554007 (+ 949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40982: BlockingRecv error: recv got EOF from 127.0.0.1:40982 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260929 20:34:59.562117 20269 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:34:59.559704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40998 (local address 127.28.244.189:38609)
0929 20:34:59.559966 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:59.559973 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:59.560541 (+ 568us) server_negotiation.cc:407] Connection header received
0929 20:34:59.560775 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:59.560783 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:59.560903 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:59.561029 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:59.561859 (+ 830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40998: BlockingRecv error: recv got EOF from 127.0.0.1:40998 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260929 20:34:59.572031 20269 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:34:59.569665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41014 (local address 127.28.244.189:38609)
0929 20:34:59.569870 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:59.569876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:59.570465 (+ 589us) server_negotiation.cc:407] Connection header received
0929 20:34:59.570659 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:59.570665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:59.570776 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:59.570894 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:59.571742 (+ 848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41014: BlockingRecv error: recv got EOF from 127.0.0.1:41014 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 20:34:59.589243 20269 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:34:59.585906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41020 (local address 127.28.244.189:38609)
0929 20:34:59.587074 (+ 1168us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:59.587081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:59.587600 (+ 519us) server_negotiation.cc:407] Connection header received
0929 20:34:59.587795 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:59.587840 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:59.587958 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:59.588075 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:59.588986 (+ 911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41020: BlockingRecv error: recv got EOF from 127.0.0.1:41020 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 20:34:59.611474 20269 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:34:59.609049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41036 (local address 127.28.244.189:38609)
0929 20:34:59.609246 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:59.609252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:59.609798 (+ 546us) server_negotiation.cc:407] Connection header received
0929 20:34:59.610032 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:59.610038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:59.610156 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:59.610289 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:59.611219 (+ 930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41036: BlockingRecv error: recv got EOF from 127.0.0.1:41036 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 20:34:59.650270 20269 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:34:59.647613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41038 (local address 127.28.244.189:38609)
0929 20:34:59.647874 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:59.647880 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:59.648426 (+ 546us) server_negotiation.cc:407] Connection header received
0929 20:34:59.648644 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:59.648651 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:59.648752 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:59.648869 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:59.649986 (+ 1117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41038: BlockingRecv error: recv got EOF from 127.0.0.1:41038 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260929 20:34:59.720098 20269 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:34:59.718044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41050 (local address 127.28.244.189:38609)
0929 20:34:59.718278 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:59.718286 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:59.718679 (+ 393us) server_negotiation.cc:407] Connection header received
0929 20:34:59.718856 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:59.718864 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:59.718966 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:59.719105 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:59.719851 (+ 746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41050: BlockingRecv error: recv got EOF from 127.0.0.1:41050 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 20:34:59.811404 20374 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6c8af3248b54a3c85fc56aa28854284 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c6c8af3248b54a3c85fc56aa28854284" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c8af3248b54a3c85fc56aa28854284" member_type: VOTER last_known_addr { host: "127.28.244.190" port: 46381 } } peers { permanent_uuid: "ad0880fdd1924c4087fb41a41efb4360" member_type: VOTER last_known_addr { host: "127.28.244.189" port: 38609 } } peers { permanent_uuid: "f70b427d6a734f019f7793b035931ae7" member_type: VOTER last_known_addr { host: "127.28.244.188" port: 37651 } } }
I20260929 20:34:59.811748 20374 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6c8af3248b54a3c85fc56aa28854284 [sys.catalog]: This master's current role is: LEADER
W20260929 20:34:59.854525 20269 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:34:59.852360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41066 (local address 127.28.244.189:38609)
0929 20:34:59.852619 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 20:34:59.852629 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:34:59.853095 (+ 466us) server_negotiation.cc:407] Connection header received
0929 20:34:59.853273 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:34:59.853277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:34:59.853352 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:34:59.853440 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:34:59.854247 (+ 807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41066: BlockingRecv error: recv got EOF from 127.0.0.1:41066 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260929 20:35:00.116178 20269 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:35:00.114038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41078 (local address 127.28.244.189:38609)
0929 20:35:00.114234 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 20:35:00.114239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:35:00.114537 (+ 298us) server_negotiation.cc:407] Connection header received
0929 20:35:00.114821 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:35:00.114826 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:35:00.114946 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:35:00.115071 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:35:00.115933 (+ 862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41078: BlockingRecv error: recv got EOF from 127.0.0.1:41078 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 20:35:00.141211 20358 catalog_manager.cc:1259] Loaded cluster ID: 2d74dce783554b82852d2a9eafd66cdc
I20260929 20:35:00.141314 20358 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f70b427d6a734f019f7793b035931ae7: loading cluster ID for follower catalog manager: success
I20260929 20:35:00.143791 20358 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f70b427d6a734f019f7793b035931ae7: acquiring CA information for follower catalog manager: success
I20260929 20:35:00.145681 20358 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f70b427d6a734f019f7793b035931ae7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 20:35:00.507676 20417 catalog_manager.cc:1259] Loaded cluster ID: 2d74dce783554b82852d2a9eafd66cdc
I20260929 20:35:00.507844 20417 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ad0880fdd1924c4087fb41a41efb4360: loading cluster ID for follower catalog manager: success
I20260929 20:35:00.510648 20417 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ad0880fdd1924c4087fb41a41efb4360: acquiring CA information for follower catalog manager: success
I20260929 20:35:00.512635 20417 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ad0880fdd1924c4087fb41a41efb4360: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 20:35:00.639441 20474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:35:00.632189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41082 (local address 127.28.244.189:38609)
0929 20:35:00.632819 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0929 20:35:00.632825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:35:00.633101 (+ 276us) server_negotiation.cc:407] Connection header received
0929 20:35:00.633280 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:35:00.633285 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:35:00.633449 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:35:00.633569 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:35:00.634070 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:35:00.634880 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:35:00.636935 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:35:00.637394 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:35:00.637491 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:35:00.638109 (+ 618us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:35:00.638718 (+ 609us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:35:00.638844 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 20:35:00.638989 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 20:35:00.639085 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":285,"threads_started":1}
I20260929 20:35:00.657469 20481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:35:00.647432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53410 (local address 127.28.244.188:37651)
0929 20:35:00.652185 (+ 4753us) server_negotiation.cc:206] Beginning negotiation
0929 20:35:00.652190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:35:00.652207 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:35:00.652307 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:35:00.652311 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:35:00.652476 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:35:00.652673 (+ 197us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:35:00.653139 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:35:00.653838 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:35:00.655171 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:35:00.655584 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:35:00.655721 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:35:00.656309 (+ 588us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:35:00.656839 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:35:00.657013 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 20:35:00.657065 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 20:35:00.657150 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4644,"thread_start_us":4532,"threads_started":1}
I20260929 20:35:00.680007 20474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:35:00.670199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41092 (local address 127.28.244.189:38609)
0929 20:35:00.670428 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 20:35:00.670439 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:35:00.672665 (+ 2226us) server_negotiation.cc:407] Connection header received
0929 20:35:00.672930 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:35:00.672937 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:35:00.673161 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:35:00.673288 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:35:00.674152 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:35:00.675310 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:35:00.677515 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:35:00.678194 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:35:00.678296 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:35:00.679142 (+ 846us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:35:00.679284 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:35:00.679348 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 20:35:00.679589 (+ 241us) server_negotiation.cc:299] Negotiation successful
0929 20:35:00.679683 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 20:35:00.683475 20487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:35:00.669876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40136 (local address 127.28.244.190:46381)
0929 20:35:00.672111 (+ 2235us) server_negotiation.cc:206] Beginning negotiation
0929 20:35:00.672118 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:35:00.673878 (+ 1760us) server_negotiation.cc:407] Connection header received
0929 20:35:00.674014 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:35:00.674020 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:35:00.674276 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:35:00.674470 (+ 194us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:35:00.675372 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:35:00.676837 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:35:00.679789 (+ 2952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:35:00.680731 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:35:00.680886 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:35:00.681654 (+ 768us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:35:00.682155 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:35:00.682277 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 20:35:00.682877 (+ 600us) server_negotiation.cc:299] Negotiation successful
0929 20:35:00.683001 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2092,"thread_start_us":1992,"threads_started":1}
I20260929 20:35:00.687917 20481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:35:00.674288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53424 (local address 127.28.244.188:37651)
0929 20:35:00.674479 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 20:35:00.674484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:35:00.681397 (+ 6913us) server_negotiation.cc:407] Connection header received
0929 20:35:00.681671 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:35:00.681677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:35:00.681789 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:35:00.681853 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:35:00.682438 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:35:00.683574 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:35:00.685916 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:35:00.686496 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:35:00.686603 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:35:00.687108 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:35:00.687233 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:35:00.687310 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 20:35:00.687471 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 20:35:00.687564 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:35:00.695042 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 20122
I20260929 20:35:00.696549 20184 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:35:00.812022 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 20122
I20260929 20:35:00.834190 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 20196
I20260929 20:35:00.835283 20259 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:35:00.953718 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 20196
I20260929 20:35:00.970798 29650 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNrwVME/build/asan/bin/kudu pid 20272
I20260929 20:35:00.972021 20334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:35:01.080600 29650 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNrwVME/build/asan/bin/kudu with pid 20272
2026-09-29T20:35:01Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4971 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9452 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (379841 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST